[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:35:46 dist-test-slave-jlnt krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:35:46 dist-test-slave-jlnt krb5kdc[399](info): set up 2 sockets
Sep 24 04:35:46 dist-test-slave-jlnt krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 04:35:47.742264 392 mini_kdc.cc:121] Time spent starting KDC: real 2.406s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:35:47 dist-test-slave-jlnt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:35:47.861471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:47.968084 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:47.968345 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:47.968386 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:47.972397 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:35:47.972457 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:47.972481 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:47.972498 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:35:47.972513 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:35:47.977684 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:47.978631 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:47.979598 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:47.988622 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:47.988713 412 server_base.cc:1034] running on GCE node
W20260924 04:35:47.988629 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:47.988675 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:47.989475 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:47.990582 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:47.991752 412 hybrid_clock.cc:648] HybridClock initialized: now 1790224547991688 us; error 78 us; skew 500 ppm
Sep 24 04:35:47 dist-test-slave-jlnt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:47.994570 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:35:47.995422 412 webserver.cc:460] Webserver started at http://127.0.98.62:34173/ using document root <none> and password file <none>
I20260924 04:35:47.995985 412 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:47.996076 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:47.996325 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:47.998430 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "7bcbc427c6414fbb95d8172916d68f3f"
format_stamp: "Formatted at 2026-09-24 04:35:47 on dist-test-slave-jlnt"
server_key: "2f935404e1604b95d2a2d8ac6263df4e"
server_key_iv: "357275d6a72b1795161dd1fd16a5530a"
server_key_version: "encryptionkey@0"
I20260924 04:35:47.999012 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "7bcbc427c6414fbb95d8172916d68f3f"
format_stamp: "Formatted at 2026-09-24 04:35:47 on dist-test-slave-jlnt"
server_key: "2f935404e1604b95d2a2d8ac6263df4e"
server_key_iv: "357275d6a72b1795161dd1fd16a5530a"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.002831 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:35:48.005367 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.006639 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:35:48.006788 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "7bcbc427c6414fbb95d8172916d68f3f"
format_stamp: "Formatted at 2026-09-24 04:35:47 on dist-test-slave-jlnt"
server_key: "2f935404e1604b95d2a2d8ac6263df4e"
server_key_iv: "357275d6a72b1795161dd1fd16a5530a"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.006903 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:48.018720 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:48.021718 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:48.021921 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:48.028715 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45503
I20260924 04:35:48.028779 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45503 every 8 connection(s)
I20260924 04:35:48.029686 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:35:48.032287 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:48.037475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 412
I20260924 04:35:48.037647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:35:48.038221 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f: Bootstrap starting.
I20260924 04:35:48.039223 392 external_mini_cluster.cc:1428] Setting key 05b97e2ecb4a61bff888f2864849f564
I20260924 04:35:48.041353 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:48.042248 480 log.cc:824] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:48.044168 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f: No bootstrap required, opened a new log
I20260924 04:35:48.048187 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcbc427c6414fbb95d8172916d68f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260924 04:35:48.048462 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:48.048532 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bcbc427c6414fbb95d8172916d68f3f, State: Initialized, Role: FOLLOWER
I20260924 04:35:48.048995 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcbc427c6414fbb95d8172916d68f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260924 04:35:48.049119 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:35:48.049227 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:35:48.049348 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:48.050248 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcbc427c6414fbb95d8172916d68f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260924 04:35:48.050578 480 leader_election.cc:302] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bcbc427c6414fbb95d8172916d68f3f; no voters:
I20260924 04:35:48.050861 480 leader_election.cc:288] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:35:48.050980 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:48.051213 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [term 1 LEADER]: Becoming Leader. State: Replica: 7bcbc427c6414fbb95d8172916d68f3f, State: Running, Role: LEADER
I20260924 04:35:48.051508 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcbc427c6414fbb95d8172916d68f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260924 04:35:48.052142 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:35:48.053166 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bcbc427c6414fbb95d8172916d68f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcbc427c6414fbb95d8172916d68f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260924 04:35:48.053295 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [sys.catalog]: This master's current role is: LEADER
I20260924 04:35:48.053506 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bcbc427c6414fbb95d8172916d68f3f. Latest consensus state: current_term: 1 leader_uuid: "7bcbc427c6414fbb95d8172916d68f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcbc427c6414fbb95d8172916d68f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260924 04:35:48.053594 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f [sys.catalog]: This master's current role is: LEADER
I20260924 04:35:48.053941 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:35:48.057415 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:48.063714 494 catalog_manager.cc:1347] Generated new cluster ID: 0defb1af32a64fe6b5e44a2f229e6d93
I20260924 04:35:48.063776 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:35:48.067425 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:48.042696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.0.98.62:45503)
0924 04:35:48.043087 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:48.043106 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:48.044411 (+ 1305us) server_negotiation.cc:407] Connection header received
0924 04:35:48.047739 (+ 3328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:48.047761 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:48.048161 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:48.048525 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:48.051149 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.052710 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.054466 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.055092 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.055285 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:48.060198 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:48.060205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:48.060226 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:48.060251 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:48.062784 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.063314 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.063322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.063332 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.063509 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.063970 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.063973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.063975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.064485 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:48.064697 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:48.066285 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0924 04:35:48.066568 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":112,"threads_started":1}
I20260924 04:35:48.095274 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:35:48.096594 494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:35:48.108194 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bcbc427c6414fbb95d8172916d68f3f: Generated new TSK 0
I20260924 04:35:48.109025 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:35:48.142306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:48.249477 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:48.249713 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:48.249755 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:48.254215 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:48.254266 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:48.254324 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:35:48.259387 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:48.260298 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:48.261235 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:48.270417 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:48.270542 506 server_base.cc:1034] running on GCE node
W20260924 04:35:48.270430 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:48.270586 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:48.271080 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:48.271690 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:48.272871 506 hybrid_clock.cc:648] HybridClock initialized: now 1790224548272856 us; error 26 us; skew 500 ppm
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:48.289067 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:35:48.289983 506 webserver.cc:460] Webserver started at http://127.0.98.1:37533/ using document root <none> and password file <none>
I20260924 04:35:48.290556 506 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:48.290654 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:48.290887 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:48.292878 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "ae93d1a6c52d42ebab013d42bc0a4a9e"
format_stamp: "Formatted at 2026-09-24 04:35:48 on dist-test-slave-jlnt"
server_key: "f3c9d22154d475cc2349fb8f97f6dd1b"
server_key_iv: "b494da28b475f69da71cf12e4e5cc338"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.293452 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ae93d1a6c52d42ebab013d42bc0a4a9e"
format_stamp: "Formatted at 2026-09-24 04:35:48 on dist-test-slave-jlnt"
server_key: "f3c9d22154d475cc2349fb8f97f6dd1b"
server_key_iv: "b494da28b475f69da71cf12e4e5cc338"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.296944 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:35:48.299295 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.300503 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:35:48.300628 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "ae93d1a6c52d42ebab013d42bc0a4a9e"
format_stamp: "Formatted at 2026-09-24 04:35:48 on dist-test-slave-jlnt"
server_key: "f3c9d22154d475cc2349fb8f97f6dd1b"
server_key_iv: "b494da28b475f69da71cf12e4e5cc338"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.300722 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:48.311587 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:48.314589 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:48.314771 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:48.315294 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:35:48.316035 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:48.316082 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.316146 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:48.316177 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.324927 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35073
I20260924 04:35:48.324960 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35073 every 8 connection(s)
I20260924 04:35:48.325788 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:35:48.327876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 506
I20260924 04:35:48.327981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:35:48.328259 392 external_mini_cluster.cc:1428] Setting key d9e3f80b7efe5fe60963d1a5bddcf731
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:35:48.339807 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:48.327581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46041 (local address 127.0.98.62:45503)
0924 04:35:48.327727 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:48.327730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:48.328674 (+ 944us) server_negotiation.cc:407] Connection header received
0924 04:35:48.330318 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:48.330321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:48.330363 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:48.330431 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:48.331841 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.332371 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.333667 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.333906 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.334029 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:48.336349 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:48.336352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:48.336354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:48.336375 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:48.337874 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.338366 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.338371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.338373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.338451 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.338779 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.338784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.338786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.338960 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:48.339066 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:48.339517 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 04:35:48.339639 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:35:48.340552 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:48.327982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45503 (local address 127.0.98.1:46041)
0924 04:35:48.328502 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:48.328542 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:48.330103 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:48.330610 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:48.330622 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:48.331120 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:48.331705 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:48.331719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.332493 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.332496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:48.333507 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:48.333516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.334050 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.334055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:48.334227 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:48.334830 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:48.334845 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:48.336188 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:48.338006 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:48.338011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:48.338024 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:48.338264 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:48.338542 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:48.338547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:48.338549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:48.338685 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:48.339062 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:48.339068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:48.339296 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 04:35:48.339478 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:35:48.339774 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":136,"threads_started":1}
I20260924 04:35:48.342545 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45503
I20260924 04:35:48.342792 635 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:48.343259 635 heartbeater.cc:511] Master 127.0.98.62:45503 requested a full tablet report, sending...
I20260924 04:35:48.344740 444 ts_manager.cc:194] Registered new tserver with Master: ae93d1a6c52d42ebab013d42bc0a4a9e (127.0.98.1:35073)
I20260924 04:35:48.345884 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:35:48.355893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:48.465420 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:48.465677 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:48.465723 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:48.469908 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:48.469960 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:48.470017 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:35:48.475093 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:48.476012 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:48.476929 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:48.486176 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:48.486202 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:48.486202 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:48.486388 640 server_base.cc:1034] running on GCE node
I20260924 04:35:48.486812 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:48.487413 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:48.488602 640 hybrid_clock.cc:648] HybridClock initialized: now 1790224548488588 us; error 28 us; skew 500 ppm
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:48.491323 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:35:48.492179 640 webserver.cc:460] Webserver started at http://127.0.98.2:38493/ using document root <none> and password file <none>
I20260924 04:35:48.492717 640 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:48.492805 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:48.493024 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:48.494982 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "3a7328473a774c3a9b5ae3d47ab025fe"
format_stamp: "Formatted at 2026-09-24 04:35:48 on dist-test-slave-jlnt"
server_key: "8aab8339f6a708a37b7af90cb49ff455"
server_key_iv: "a414896eb4943b460a5043e5f0102fe0"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.495541 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3a7328473a774c3a9b5ae3d47ab025fe"
format_stamp: "Formatted at 2026-09-24 04:35:48 on dist-test-slave-jlnt"
server_key: "8aab8339f6a708a37b7af90cb49ff455"
server_key_iv: "a414896eb4943b460a5043e5f0102fe0"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.499472 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:35:48.501777 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.502926 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:35:48.503057 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "3a7328473a774c3a9b5ae3d47ab025fe"
format_stamp: "Formatted at 2026-09-24 04:35:48 on dist-test-slave-jlnt"
server_key: "8aab8339f6a708a37b7af90cb49ff455"
server_key_iv: "a414896eb4943b460a5043e5f0102fe0"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.503170 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:48.517447 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:48.520109 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:48.520293 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:48.520810 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:35:48.521622 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:48.521692 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.521754 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:48.521798 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.531154 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35659
I20260924 04:35:48.531180 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35659 every 8 connection(s)
I20260924 04:35:48.532003 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:48.541646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 640
I20260924 04:35:48.541772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:35:48.542096 392 external_mini_cluster.cc:1428] Setting key a081a913dc8d22895150d3269eb5de7f
I20260924 04:35:48.545360 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:48.533730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58235 (local address 127.0.98.62:45503)
0924 04:35:48.533856 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:48.533860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:48.534647 (+ 787us) server_negotiation.cc:407] Connection header received
0924 04:35:48.535973 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:48.535976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:48.536033 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:48.536137 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:48.537245 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.537784 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.538883 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.539105 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.539220 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:48.541603 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:48.541606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:48.541608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:48.541630 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:48.543299 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.543872 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.543877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.543879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.543964 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.544286 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.544290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.544292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.544468 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:48.544591 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:48.545054 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 04:35:48.545197 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:35:48.546171 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:48.534089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45503 (local address 127.0.98.2:58235)
0924 04:35:48.534498 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:48.534538 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:48.535751 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:48.536298 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:48.536305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:48.536630 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:48.537076 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:48.537089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.537907 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.537910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:48.538760 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:48.538768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.539249 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.539253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:48.539414 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:48.539976 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:48.539992 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:48.541408 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:48.543412 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:48.543419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:48.543438 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:48.543763 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:48.544060 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:48.544063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:48.544065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:48.544192 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:48.544590 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:48.544596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:48.544826 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 04:35:48.545001 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 04:35:48.545329 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260924 04:35:48.547361 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45503
I20260924 04:35:48.547623 769 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:48.548105 769 heartbeater.cc:511] Master 127.0.98.62:45503 requested a full tablet report, sending...
I20260924 04:35:48.549358 444 ts_manager.cc:194] Registered new tserver with Master: 3a7328473a774c3a9b5ae3d47ab025fe (127.0.98.2:35659)
I20260924 04:35:48.550164 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58235
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:35:48.572328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:48.681090 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:48.681404 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:48.681475 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:48.685659 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:48.685735 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:48.685818 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:35:48.690933 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:48.691867 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:48.692816 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:48.701864 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:48.702051 774 server_base.cc:1034] running on GCE node
W20260924 04:35:48.701942 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:48.701867 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:48.702550 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:48.703120 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:48.704312 774 hybrid_clock.cc:648] HybridClock initialized: now 1790224548704279 us; error 46 us; skew 500 ppm
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:48.707207 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:35:48.707998 774 webserver.cc:460] Webserver started at http://127.0.98.3:33117/ using document root <none> and password file <none>
I20260924 04:35:48.708524 774 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:48.708581 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:48.708807 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:48.710891 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4b5706060cf49b69651a6abeda86ad7"
format_stamp: "Formatted at 2026-09-24 04:35:48 on dist-test-slave-jlnt"
server_key: "7d78b06291bfe6e165879a50414645d2"
server_key_iv: "2eaeceea4674b695366e1b9406939933"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.711472 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4b5706060cf49b69651a6abeda86ad7"
format_stamp: "Formatted at 2026-09-24 04:35:48 on dist-test-slave-jlnt"
server_key: "7d78b06291bfe6e165879a50414645d2"
server_key_iv: "2eaeceea4674b695366e1b9406939933"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.715054 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:35:48.717363 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.718494 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:35:48.718621 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "d4b5706060cf49b69651a6abeda86ad7"
format_stamp: "Formatted at 2026-09-24 04:35:48 on dist-test-slave-jlnt"
server_key: "7d78b06291bfe6e165879a50414645d2"
server_key_iv: "2eaeceea4674b695366e1b9406939933"
server_key_version: "encryptionkey@0"
I20260924 04:35:48.718727 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:48.732759 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:48.735569 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:48.735759 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:48.736302 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:35:48.737066 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:48.737133 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.737262 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:48.737316 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:48.746573 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41115
I20260924 04:35:48.746641 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41115 every 8 connection(s)
I20260924 04:35:48.747467 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:48.757730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 774
I20260924 04:35:48.757860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:35:48.758133 392 external_mini_cluster.cc:1428] Setting key 57529a48bb95cccb4fadb07a6b6c6ff8
I20260924 04:35:48.761196 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:48.749260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40645 (local address 127.0.98.62:45503)
0924 04:35:48.749401 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:48.749405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:48.750251 (+ 846us) server_negotiation.cc:407] Connection header received
0924 04:35:48.751512 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:48.751515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:48.751564 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:48.751644 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:48.752957 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.753567 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.754751 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.754962 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.755045 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:48.757422 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:48.757425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:48.757427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:48.757450 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:48.759039 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.759685 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.759688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.759689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.759763 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.760109 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.760112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.760114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.760329 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:48.760433 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:48.760940 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 04:35:48.761032 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:35:48.761991 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:48.749658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45503 (local address 127.0.98.3:40645)
0924 04:35:48.750096 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:48.750136 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:48.751309 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:48.751793 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:48.751799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:48.752146 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:48.752792 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:48.752828 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.753668 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.753670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:48.754641 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:48.754650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.755089 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.755091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:48.755221 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:48.755760 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:48.755776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:48.757278 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:48.759172 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:48.759181 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:48.759192 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:48.759548 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:48.759862 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:48.759866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:48.759868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:48.760009 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:48.760436 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:48.760441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:48.760703 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 04:35:48.760883 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 04:35:48.761115 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260924 04:35:48.763096 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45503
I20260924 04:35:48.763288 903 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:48.763682 903 heartbeater.cc:511] Master 127.0.98.62:45503 requested a full tablet report, sending...
I20260924 04:35:48.764590 444 ts_manager.cc:194] Registered new tserver with Master: d4b5706060cf49b69651a6abeda86ad7 (127.0.98.3:41115)
I20260924 04:35:48.765094 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40645
I20260924 04:35:48.772343 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:35:48.798069 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:48.788231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.0.98.1:35073)
0924 04:35:48.788550 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:48.788566 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:48.788621 (+ 55us) server_negotiation.cc:407] Connection header received
0924 04:35:48.788787 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:48.788795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:48.788983 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:48.789179 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:48.789840 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.790994 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.791826 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.792192 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.792305 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:48.794720 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:48.794728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:48.794741 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:48.794764 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:48.796222 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.796652 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.796657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.796658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.796736 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.797001 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.797006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.797008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.797297 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:48.797498 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:48.797744 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 04:35:48.797887 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
W20260924 04:35:48.798982 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33844
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:35:48.823392 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:48.815633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.0.98.1:35073)
0924 04:35:48.815775 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:48.815778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:48.815792 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:35:48.815915 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:48.815918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:48.815955 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:48.816023 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:48.816657 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.817195 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.817969 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:48.818183 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:48.818256 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:48.820468 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:48.820471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:48.820473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:48.820493 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:48.821930 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.822300 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.822303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.822304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.822371 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:48.822604 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:48.822606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:48.822607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:48.822747 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:48.822883 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:48.823119 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 04:35:48.823236 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:35:48.824323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 506
I20260924 04:35:48.830027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 640
I20260924 04:35:48.837240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 774
I20260924 04:35:48.842583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 412
2026-09-24T04:35:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3537 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[917](info): set up 2 sockets
Sep 24 04:35:48 dist-test-slave-jlnt krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 04:35:49.876778 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:35:49 dist-test-slave-jlnt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:35:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:35:49.996143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43475
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:50.102932 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:50.103216 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:50.103294 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:50.107261 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:35:50.107340 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:50.107378 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:50.107436 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:35:50.107475 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:35:50.112511 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:50.113554 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:50.114527 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:50.123238 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:50.123220 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:50.123387 930 server_base.cc:1034] running on GCE node
W20260924 04:35:50.123252 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:50.124156 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:50.125272 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:50.126458 930 hybrid_clock.cc:648] HybridClock initialized: now 1790224550126436 us; error 30 us; skew 500 ppm
Sep 24 04:35:50 dist-test-slave-jlnt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:50.129348 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:35:50.130298 930 webserver.cc:460] Webserver started at http://127.0.98.62:35219/ using document root <none> and password file <none>
I20260924 04:35:50.130848 930 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:50.130910 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:50.131127 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:50.133204 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "e22215a89af23752b57c92774d554570"
server_key_iv: "32b7fc691bc0c1725c2535c08dea287d"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.133759 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "e22215a89af23752b57c92774d554570"
server_key_iv: "32b7fc691bc0c1725c2535c08dea287d"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.137409 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:35:50.139813 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.141101 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:35:50.141312 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "e22215a89af23752b57c92774d554570"
server_key_iv: "32b7fc691bc0c1725c2535c08dea287d"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.141429 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:50.154546 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:50.157436 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:50.157636 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:50.164436 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43475
I20260924 04:35:50.164491 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43475 every 8 connection(s)
I20260924 04:35:50.165450 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:35:50.168089 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:50.171854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 930
I20260924 04:35:50.171991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:35:50.172238 392 external_mini_cluster.cc:1428] Setting key c8083f82b0d81d789f56b85d677f6f5a
I20260924 04:35:50.173732 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd: Bootstrap starting.
I20260924 04:35:50.176100 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:50.176820 998 log.cc:824] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:50.178579 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd: No bootstrap required, opened a new log
Sep 24 04:35:50 dist-test-slave-jlnt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:50.182372 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } }
I20260924 04:35:50.182587 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:50.182644 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed27af8beb7b4f5a9b0785bb652ac8cd, State: Initialized, Role: FOLLOWER
I20260924 04:35:50.183115 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } }
I20260924 04:35:50.183233 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:35:50.183319 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:35:50.183432 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:50.184458 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } }
I20260924 04:35:50.184824 998 leader_election.cc:302] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed27af8beb7b4f5a9b0785bb652ac8cd; no voters:
I20260924 04:35:50.185129 998 leader_election.cc:288] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:35:50.185258 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:50.185490 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [term 1 LEADER]: Becoming Leader. State: Replica: ed27af8beb7b4f5a9b0785bb652ac8cd, State: Running, Role: LEADER
I20260924 04:35:50.185806 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } }
I20260924 04:35:50.186354 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:35:50.186985 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed27af8beb7b4f5a9b0785bb652ac8cd. Latest consensus state: current_term: 1 leader_uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } } }
I20260924 04:35:50.187086 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:35:50.187417 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed27af8beb7b4f5a9b0785bb652ac8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } } }
I20260924 04:35:50.187461 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.173640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.0.98.62:43475)
0924 04:35:50.174056 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.174071 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.174105 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:35:50.174714 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.174730 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.175090 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.175364 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:50.176271 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.177387 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.178250 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.178757 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.178916 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.181754 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:50.181760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:50.181772 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:50.181797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:50.183843 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.184318 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.184322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.184326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.184431 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.184735 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.184737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.184738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.185219 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:50.186015 (+ 796us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.186172 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.186466 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":155,"threads_started":1}
I20260924 04:35:50.187505 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:35:50.187942 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:35:50.191272 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:35:50.196749 1010 catalog_manager.cc:1347] Generated new cluster ID: 57fc4dc029284ea5a5afe00b7429d573
I20260924 04:35:50.196816 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:35:50.202509 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:35:50.203382 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:35:50.210456 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed27af8beb7b4f5a9b0785bb652ac8cd: Generated new TSK 0
I20260924 04:35:50.211297 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:35:50.239741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:50.348975 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:50.349278 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:50.349354 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:50.353413 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:50.353497 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:50.353585 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:35:50.358692 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:50.359719 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:50.360743 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:35:50.370281 1024 server_base.cc:1034] running on GCE node
W20260924 04:35:50.370217 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:50.370240 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:50.370261 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:50.370910 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:50.371524 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:50.372728 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790224550372719 us; error 33 us; skew 500 ppm
Sep 24 04:35:50 dist-test-slave-jlnt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:50.375546 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:35:50.376410 1024 webserver.cc:460] Webserver started at http://127.0.98.1:35755/ using document root <none> and password file <none>
I20260924 04:35:50.376963 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:50.377025 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:50.377293 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:50.379307 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "d8efd3b5f46149a599f3fbd1576c5ce2"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "90a08bc86846ac0ae36b28bb042dd1d1"
server_key_iv: "1267fe0533d4364ecc3abce16fc34e77"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.379871 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d8efd3b5f46149a599f3fbd1576c5ce2"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "90a08bc86846ac0ae36b28bb042dd1d1"
server_key_iv: "1267fe0533d4364ecc3abce16fc34e77"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.383533 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:35:50.385993 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.387267 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:35:50.387379 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "d8efd3b5f46149a599f3fbd1576c5ce2"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "90a08bc86846ac0ae36b28bb042dd1d1"
server_key_iv: "1267fe0533d4364ecc3abce16fc34e77"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.387480 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:50.409878 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:50.412726 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:50.412925 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:50.413530 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:35:50.414335 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:50.414381 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.414446 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:50.414475 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.423596 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42191
I20260924 04:35:50.423621 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42191 every 8 connection(s)
I20260924 04:35:50.424490 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:35:50.426056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 1024
I20260924 04:35:50.426172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:35:50.426470 392 external_mini_cluster.cc:1428] Setting key ba8aa1e2426c8620c94102912e07fbfb
Sep 24 04:35:50 dist-test-slave-jlnt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:35:50.438973 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.426403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38967 (local address 127.0.98.62:43475)
0924 04:35:50.426591 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.426596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.427575 (+ 979us) server_negotiation.cc:407] Connection header received
0924 04:35:50.429093 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.429097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.429183 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.429259 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:50.430412 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.430942 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.432077 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.432495 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.432610 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.435115 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:50.435119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:50.435129 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:50.435158 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:50.436897 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.437566 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.437573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.437577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.437677 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.438015 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.438018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.438019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.438196 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:50.438274 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.438759 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.438853 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:35:50.439827 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.426885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43475 (local address 127.0.98.1:38967)
0924 04:35:50.427431 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:50.427487 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:50.428889 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:50.429428 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:50.429436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:50.429811 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:50.430271 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.430284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.431082 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.431084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.431951 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.431959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.432658 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.432661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.432818 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.433533 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:50.433551 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:50.434951 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:50.437026 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:50.437031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:50.437042 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:50.437464 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:50.437785 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:50.437787 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:50.437788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:50.437917 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:50.438286 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:50.438290 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:50.438509 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 04:35:50.438724 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 04:35:50.438972 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:35:50.441097 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43475
I20260924 04:35:50.441381 1153 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:50.441977 1153 heartbeater.cc:511] Master 127.0.98.62:43475 requested a full tablet report, sending...
I20260924 04:35:50.443676 962 ts_manager.cc:194] Registered new tserver with Master: d8efd3b5f46149a599f3fbd1576c5ce2 (127.0.98.1:42191)
I20260924 04:35:50.444978 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:35:50.456494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:50.565037 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:50.565322 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:50.565421 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:50.569491 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:50.569553 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:50.569660 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:35:50.574805 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:50.575820 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:50.576844 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:50.586007 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:50.586002 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:50.586002 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:50.586252 1158 server_base.cc:1034] running on GCE node
I20260924 04:35:50.586694 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:50.587239 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:50.588421 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790224550588386 us; error 44 us; skew 500 ppm
Sep 24 04:35:50 dist-test-slave-jlnt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:50.591167 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:35:50.592097 1158 webserver.cc:460] Webserver started at http://127.0.98.2:36997/ using document root <none> and password file <none>
I20260924 04:35:50.592643 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:50.592716 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:50.592940 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:50.595028 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "9871feca41ce4b3aa6a94bc39145896c"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "ab86d45e55a38c56596933b9674ac80e"
server_key_iv: "fa1c5bd7e4f97a866e2becce33abe05e"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.595579 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9871feca41ce4b3aa6a94bc39145896c"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "ab86d45e55a38c56596933b9674ac80e"
server_key_iv: "fa1c5bd7e4f97a866e2becce33abe05e"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.599198 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:35:50.601999 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.603155 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:35:50.603253 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "9871feca41ce4b3aa6a94bc39145896c"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "ab86d45e55a38c56596933b9674ac80e"
server_key_iv: "fa1c5bd7e4f97a866e2becce33abe05e"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.603385 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:50.625355 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:50.628755 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:50.628966 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:50.629639 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:35:50.630438 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:50.630484 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.630522 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:50.630538 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.639153 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44691
I20260924 04:35:50.639178 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44691 every 8 connection(s)
I20260924 04:35:50.640050 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:35:50.642424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 1158
I20260924 04:35:50.642581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:35:50.642886 392 external_mini_cluster.cc:1428] Setting key 81acfe747f89a67c734319934d60e224
Sep 24 04:35:50 dist-test-slave-jlnt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:35:50.656457 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.641800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52083 (local address 127.0.98.62:43475)
0924 04:35:50.641947 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.641958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.642951 (+ 993us) server_negotiation.cc:407] Connection header received
0924 04:35:50.644695 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.644697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.644738 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.644809 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:50.646439 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.647345 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.649208 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.649437 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.649537 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.652438 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:50.652442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:50.652444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:50.652473 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:50.654019 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.654749 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.654755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.654757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.654846 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.655289 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.655293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.655295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.655481 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:50.655631 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.656166 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.656281 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:35:50.657373 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.642235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43475 (local address 127.0.98.2:52083)
0924 04:35:50.642766 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:50.642825 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:50.644460 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:50.644998 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:50.645009 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:50.645555 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:50.646258 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.646281 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.647512 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.647519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.649034 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.649048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.649606 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.649613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.649850 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.650494 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:50.650523 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:50.652248 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:50.654157 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:50.654163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:50.654180 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:50.654439 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:50.654951 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:50.654954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:50.654956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:50.655182 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:50.655607 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:50.655613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:50.655903 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 04:35:50.656113 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 04:35:50.656432 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":114,"threads_started":1}
I20260924 04:35:50.658458 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43475
I20260924 04:35:50.658725 1287 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:50.659282 1287 heartbeater.cc:511] Master 127.0.98.62:43475 requested a full tablet report, sending...
I20260924 04:35:50.660418 962 ts_manager.cc:194] Registered new tserver with Master: 9871feca41ce4b3aa6a94bc39145896c (127.0.98.2:44691)
I20260924 04:35:50.660971 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:35:50.672223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:50.781685 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:50.781935 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:50.781978 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:50.785929 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:50.785983 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:50.786092 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:35:50.791447 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:50.792510 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:50.793519 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:50.802876 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:50.802892 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:50.802939 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:50.803373 1292 server_base.cc:1034] running on GCE node
I20260924 04:35:50.803768 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:50.804378 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:50.805526 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790224550805514 us; error 27 us; skew 500 ppm
Sep 24 04:35:50 dist-test-slave-jlnt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:50.808271 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:35:50.809067 1292 webserver.cc:460] Webserver started at http://127.0.98.3:34085/ using document root <none> and password file <none>
I20260924 04:35:50.809686 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:50.809751 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:50.809906 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:50.811862 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbd988fdb2494d64a55be244e0492d7e"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "dfac18776e271f233306c76d1ba4dcb3"
server_key_iv: "04fdbe1ff1380a33b5f255c8fc544d48"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.812366 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbd988fdb2494d64a55be244e0492d7e"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "dfac18776e271f233306c76d1ba4dcb3"
server_key_iv: "04fdbe1ff1380a33b5f255c8fc544d48"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.816013 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:35:50.818387 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.819569 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:35:50.819669 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "dbd988fdb2494d64a55be244e0492d7e"
format_stamp: "Formatted at 2026-09-24 04:35:50 on dist-test-slave-jlnt"
server_key: "dfac18776e271f233306c76d1ba4dcb3"
server_key_iv: "04fdbe1ff1380a33b5f255c8fc544d48"
server_key_version: "encryptionkey@0"
I20260924 04:35:50.819751 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:50.846261 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:50.849469 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:50.849632 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:50.850154 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:35:50.850911 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:50.850956 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.850996 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:50.851011 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:50.859735 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36793
I20260924 04:35:50.859761 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36793 every 8 connection(s)
I20260924 04:35:50.860591 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:35:50.868350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 1292
I20260924 04:35:50.868474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:35:50.868808 392 external_mini_cluster.cc:1428] Setting key f586325d440d3509192ced47318ef699
Sep 24 04:35:50 dist-test-slave-jlnt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:50.874257 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.862331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60671 (local address 127.0.98.62:43475)
0924 04:35:50.862476 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.862480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.863286 (+ 806us) server_negotiation.cc:407] Connection header received
0924 04:35:50.864595 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.864598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.864651 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.864769 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:50.865903 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.866436 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.867572 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.867793 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.867901 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.870994 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:50.870997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:50.871005 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:50.871030 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:50.872224 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.872724 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.872728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.872730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.872811 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.873194 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.873197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.873199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.873372 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:50.873479 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.873983 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.874102 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:35:50.875075 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.862696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43475 (local address 127.0.98.3:60671)
0924 04:35:50.863135 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:50.863183 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:50.864387 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:50.864939 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:50.864945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:50.865304 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:50.865757 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.865769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.866557 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.866561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.867434 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.867443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.867935 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.867939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.868136 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.868828 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:50.868850 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:50.870840 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:50.872338 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:50.872345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:50.872360 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:50.872616 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:50.872909 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:50.872912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:50.872914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:50.873056 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:50.873487 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:50.873496 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:50.873744 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 04:35:50.873932 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 04:35:50.874270 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
I20260924 04:35:50.876194 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43475
I20260924 04:35:50.876441 1421 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:50.876946 1421 heartbeater.cc:511] Master 127.0.98.62:43475 requested a full tablet report, sending...
I20260924 04:35:50.877982 962 ts_manager.cc:194] Registered new tserver with Master: dbd988fdb2494d64a55be244e0492d7e (127.0.98.3:36793)
I20260924 04:35:50.878499 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60671
I20260924 04:35:50.882773 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:35:50.893474 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.886105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.0.98.62:43475)
0924 04:35:50.886220 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.886224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.886424 (+ 200us) server_negotiation.cc:407] Connection header received
0924 04:35:50.886576 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.886579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.886620 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.886704 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:50.887499 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.888010 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.888883 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.889086 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.889224 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.890525 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:50.890528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:50.890530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:50.890552 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:50.891938 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.892390 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.892392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.892394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.892466 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.892684 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.892686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.892687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.892865 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:50.892960 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.893124 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.893271 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:35:50.897938 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:35:50.900792 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:35:50.915102 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.909560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.98.2:44691)
0924 04:35:50.910113 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.910126 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.910662 (+ 536us) server_negotiation.cc:407] Connection header received
0924 04:35:50.910781 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.910788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.910998 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.911159 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:50.911393 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.912229 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.913695 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.914427 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.914548 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.914707 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.914850 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.914939 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":175,"threads_started":1}
I20260924 04:35:50.915092 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.909400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44691 (local address 127.0.0.1:50794)
0924 04:35:50.910564 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:50.910578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:50.910673 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:50.911138 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:50.911148 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:50.911165 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:50.911268 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.911278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.912354 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.912361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.913569 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.913578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.914547 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.914550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.914653 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.914689 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:35:50.914824 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 04:35:50.914938 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":54,"thread_start_us":115,"threads_started":1}
I20260924 04:35:50.915480 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.909257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42191 (local address 127.0.0.1:47350)
0924 04:35:50.910077 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:50.910128 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:50.910291 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:50.910949 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:50.910955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:50.910981 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:50.911130 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.911137 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.912781 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.912787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.913968 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.913976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.915071 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.915074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.915178 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.915210 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 04:35:50.915264 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:35:50.915331 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":25,"thread_start_us":86,"threads_started":1}
I20260924 04:35:50.915692 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.910006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36793 (local address 127.0.0.1:37158)
0924 04:35:50.910844 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:50.910855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:50.910920 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:50.911422 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:50.911427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:50.911448 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:50.911526 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.911531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.912847 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.912850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.914071 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.914081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.915297 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.915300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.915374 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.915394 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:35:50.915467 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:35:50.915539 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":34,"thread_start_us":98,"threads_started":1}
I20260924 04:35:50.916028 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.910572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.0.98.3:36793)
0924 04:35:50.910964 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.910975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.910994 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:35:50.911059 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.911064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.911257 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.911390 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:50.911646 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.912723 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.914202 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.914934 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.915046 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.915174 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.915522 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.915628 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":122,"threads_started":1}
I20260924 04:35:50.916210 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.909386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.0.98.1:42191)
0924 04:35:50.910302 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.910317 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.910343 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:35:50.910534 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.910540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.910770 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.911281 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:50.911443 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.912626 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.914092 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.914931 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.915735 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.915883 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.915939 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.916049 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":74,"threads_started":1}
I20260924 04:35:50.918692 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 932f77c40c7f438c89cd3010680f6bdf (DEFAULT_TABLE table=test-table [id=b0a579a0894741318127500b064d5238]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:50.918892 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 932f77c40c7f438c89cd3010680f6bdf (DEFAULT_TABLE table=test-table [id=b0a579a0894741318127500b064d5238]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:50.918895 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 932f77c40c7f438c89cd3010680f6bdf (DEFAULT_TABLE table=test-table [id=b0a579a0894741318127500b064d5238]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:50.919770 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932f77c40c7f438c89cd3010680f6bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:50.919894 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932f77c40c7f438c89cd3010680f6bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:50.919894 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932f77c40c7f438c89cd3010680f6bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:50.926339 1438 tablet_bootstrap.cc:492] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e: Bootstrap starting.
I20260924 04:35:50.927706 1439 tablet_bootstrap.cc:492] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c: Bootstrap starting.
I20260924 04:35:50.928462 1438 tablet_bootstrap.cc:654] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:50.929487 1440 tablet_bootstrap.cc:492] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2: Bootstrap starting.
I20260924 04:35:50.929543 1439 tablet_bootstrap.cc:654] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:50.929631 1438 log.cc:824] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:50.930760 1439 log.cc:824] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:50.931542 1440 tablet_bootstrap.cc:654] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:50.932287 1438 tablet_bootstrap.cc:492] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e: No bootstrap required, opened a new log
I20260924 04:35:50.932479 1438 ts_tablet_manager.cc:1397] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:35:50.932727 1440 log.cc:824] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:50.933341 1439 tablet_bootstrap.cc:492] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c: No bootstrap required, opened a new log
I20260924 04:35:50.933560 1439 ts_tablet_manager.cc:1397] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:35:50.934652 1440 tablet_bootstrap.cc:492] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2: No bootstrap required, opened a new log
I20260924 04:35:50.934827 1440 ts_tablet_manager.cc:1397] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:35:50.936023 1438 raft_consensus.cc:348] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } }
I20260924 04:35:50.936285 1438 raft_consensus.cc:374] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:50.936338 1438 raft_consensus.cc:729] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd988fdb2494d64a55be244e0492d7e, State: Initialized, Role: FOLLOWER
I20260924 04:35:50.936859 1438 consensus_queue.cc:260] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } }
I20260924 04:35:50.937273 1439 raft_consensus.cc:348] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } }
I20260924 04:35:50.937541 1439 raft_consensus.cc:374] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:50.937595 1439 raft_consensus.cc:729] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9871feca41ce4b3aa6a94bc39145896c, State: Initialized, Role: FOLLOWER
I20260924 04:35:50.937882 1438 ts_tablet_manager.cc:1428] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260924 04:35:50.938004 1439 consensus_queue.cc:260] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } }
I20260924 04:35:50.938267 1421 heartbeater.cc:503] Master 127.0.98.62:43475 was elected leader, sending a full tablet report...
I20260924 04:35:50.938484 1440 raft_consensus.cc:348] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } }
I20260924 04:35:50.938726 1440 raft_consensus.cc:374] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:50.938787 1440 raft_consensus.cc:729] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8efd3b5f46149a599f3fbd1576c5ce2, State: Initialized, Role: FOLLOWER
I20260924 04:35:50.939172 1440 consensus_queue.cc:260] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } }
I20260924 04:35:50.939675 1287 heartbeater.cc:503] Master 127.0.98.62:43475 was elected leader, sending a full tablet report...
I20260924 04:35:50.940004 1440 ts_tablet_manager.cc:1428] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:35:50.940325 1153 heartbeater.cc:503] Master 127.0.98.62:43475 was elected leader, sending a full tablet report...
I20260924 04:35:50.940871 1439 ts_tablet_manager.cc:1428] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:35:50.960323 1444 raft_consensus.cc:482] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:35:50.960539 1444 raft_consensus.cc:504] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } }
I20260924 04:35:50.961809 1444 leader_election.cc:288] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8efd3b5f46149a599f3fbd1576c5ce2 (127.0.98.1:42191), 9871feca41ce4b3aa6a94bc39145896c (127.0.98.2:44691)
I20260924 04:35:50.965251 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.962074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58803 (local address 127.0.98.1:42191)
0924 04:35:50.962194 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.962199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.962291 (+ 92us) server_negotiation.cc:407] Connection header received
0924 04:35:50.962372 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.962375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.962424 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.962509 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:50.962774 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.963274 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.964297 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.964678 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.964795 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.964857 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.965032 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.965107 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:35:50.965252 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.961994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42191 (local address 127.0.98.3:58803)
0924 04:35:50.962174 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:50.962189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:50.962285 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:50.962521 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:50.962524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:50.962539 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:50.962662 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.962667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.963385 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.963390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.964171 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.964180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.964795 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.964799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.964911 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.964928 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:35:50.965016 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:35:50.965083 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 04:35:50.965631 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.962153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47081 (local address 127.0.98.2:44691)
0924 04:35:50.962283 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.962287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.962481 (+ 194us) server_negotiation.cc:407] Connection header received
0924 04:35:50.962634 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.962637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.962681 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.962767 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:50.963020 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.963493 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.964545 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.965008 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.965123 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.965210 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.965439 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.965518 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:35:50.965932 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932f77c40c7f438c89cd3010680f6bdf" candidate_uuid: "dbd988fdb2494d64a55be244e0492d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" is_pre_election: true
I20260924 04:35:50.966007 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.962014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44691 (local address 127.0.98.3:47081)
0924 04:35:50.962389 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:50.962400 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:50.962518 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:50.962773 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:50.962776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:50.962790 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:50.962915 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.962920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.963600 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.963604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.964424 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:50.964432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.965124 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.965128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:50.965263 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.965289 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:35:50.965818 (+ 529us) client_negotiation.cc:240] Negotiation successful
0924 04:35:50.965896 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":12,"thread_start_us":99,"threads_started":1}
I20260924 04:35:50.966250 1107 raft_consensus.cc:2454] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbd988fdb2494d64a55be244e0492d7e in term 0.
I20260924 04:35:50.966634 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932f77c40c7f438c89cd3010680f6bdf" candidate_uuid: "dbd988fdb2494d64a55be244e0492d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9871feca41ce4b3aa6a94bc39145896c" is_pre_election: true
I20260924 04:35:50.966737 1308 leader_election.cc:302] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8efd3b5f46149a599f3fbd1576c5ce2, dbd988fdb2494d64a55be244e0492d7e; no voters:
I20260924 04:35:50.966933 1241 raft_consensus.cc:2454] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbd988fdb2494d64a55be244e0492d7e in term 0.
I20260924 04:35:50.967083 1444 raft_consensus.cc:2781] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:35:50.967183 1444 raft_consensus.cc:482] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:35:50.967236 1444 raft_consensus.cc:3037] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:50.968423 1444 raft_consensus.cc:504] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } }
I20260924 04:35:50.968811 1444 leader_election.cc:288] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [CANDIDATE]: Term 1 election: Requested vote from peers d8efd3b5f46149a599f3fbd1576c5ce2 (127.0.98.1:42191), 9871feca41ce4b3aa6a94bc39145896c (127.0.98.2:44691)
I20260924 04:35:50.969089 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932f77c40c7f438c89cd3010680f6bdf" candidate_uuid: "dbd988fdb2494d64a55be244e0492d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2"
I20260924 04:35:50.969291 1107 raft_consensus.cc:3037] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:50.969316 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932f77c40c7f438c89cd3010680f6bdf" candidate_uuid: "dbd988fdb2494d64a55be244e0492d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9871feca41ce4b3aa6a94bc39145896c"
I20260924 04:35:50.969478 1241 raft_consensus.cc:3037] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:50.970360 1107 raft_consensus.cc:2454] T 932f77c40c7f438c89cd3010680f6bdf P d8efd3b5f46149a599f3fbd1576c5ce2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbd988fdb2494d64a55be244e0492d7e in term 1.
I20260924 04:35:50.970595 1241 raft_consensus.cc:2454] T 932f77c40c7f438c89cd3010680f6bdf P 9871feca41ce4b3aa6a94bc39145896c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbd988fdb2494d64a55be244e0492d7e in term 1.
I20260924 04:35:50.970642 1308 leader_election.cc:302] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8efd3b5f46149a599f3fbd1576c5ce2, dbd988fdb2494d64a55be244e0492d7e; no voters:
I20260924 04:35:50.970800 1444 raft_consensus.cc:2781] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:50.971033 1444 raft_consensus.cc:686] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [term 1 LEADER]: Becoming Leader. State: Replica: dbd988fdb2494d64a55be244e0492d7e, State: Running, Role: LEADER
I20260924 04:35:50.971380 1444 consensus_queue.cc:237] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } }
I20260924 04:35:50.974452 962 catalog_manager.cc:5495] T 932f77c40c7f438c89cd3010680f6bdf P dbd988fdb2494d64a55be244e0492d7e reported cstate change: term changed from 0 to 1, leader changed from <none> to dbd988fdb2494d64a55be244e0492d7e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dbd988fdb2494d64a55be244e0492d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8efd3b5f46149a599f3fbd1576c5ce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9871feca41ce4b3aa6a94bc39145896c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbd988fdb2494d64a55be244e0492d7e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36793 } health_report { overall_health: HEALTHY } } }
Sep 24 04:35:50 dist-test-slave-jlnt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:35:50.994757 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:50.985699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.0.98.1:42191)
0924 04:35:50.985842 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:50.985847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:50.986017 (+ 170us) server_negotiation.cc:407] Connection header received
0924 04:35:50.986130 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:50.986133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:50.986246 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:50.986323 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:50.986956 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.987594 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.988445 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:50.988725 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:50.988833 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:50.991388 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:50.991397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:50.991412 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:50.991435 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:50.993126 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.993577 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.993582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.993583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.993677 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:50.993902 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:50.993904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:50.993905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:50.994132 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:50.994277 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:50.994426 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 04:35:50.994519 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 04:35:51 dist-test-slave-jlnt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:35:51 dist-test-slave-jlnt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:35:51.021281 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:51.012532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.0.98.1:42191)
0924 04:35:51.012674 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:51.012681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:51.012828 (+ 147us) server_negotiation.cc:407] Connection header received
0924 04:35:51.012937 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:51.012939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:51.012982 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:51.013056 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:51.013818 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:51.014344 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:51.015156 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:51.015381 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:51.015466 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:51.017975 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:51.017978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:51.017980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:51.018003 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:51.019668 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:51.020111 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:51.020114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:51.020115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:51.020195 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:51.020431 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:51.020433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:51.020434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:51.020593 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:51.020709 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:51.020965 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 04:35:51.021064 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 04:35:51 dist-test-slave-jlnt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:35:51 dist-test-slave-jlnt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:35:51.047233 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:51.038638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.0.98.1:42191)
0924 04:35:51.038803 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:51.038808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:51.038969 (+ 161us) server_negotiation.cc:407] Connection header received
0924 04:35:51.039153 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:51.039156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:51.039198 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:51.039283 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:51.039961 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:51.040462 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:51.041379 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:51.041597 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:51.041706 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:51.044077 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:51.044080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:51.044082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:51.044104 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:51.045638 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:51.046049 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:51.046053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:51.046055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:51.046133 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:51.046416 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:51.046420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:51.046422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:51.046583 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:51.046715 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:51.046947 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 04:35:51.047060 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 04:35:51.048491 1067 tablet_service.cc:2908] Rejecting scan request for tablet 932f77c40c7f438c89cd3010680f6bdf: Uninitialized: clean time has not yet been initialized
I20260924 04:35:51.050495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 1024
I20260924 04:35:51.056401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 1158
I20260924 04:35:51.061945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 1292
I20260924 04:35:51.067416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 930
2026-09-24T04:35:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2225 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:35:51 dist-test-slave-jlnt krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:35:51 dist-test-slave-jlnt krb5kdc[1462](info): set up 2 sockets
Sep 24 04:35:51 dist-test-slave-jlnt krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260924 04:35:52.108860 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:35:52 dist-test-slave-jlnt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:35:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:35:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:35:52.230626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:52.338042 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:52.338351 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:52.338425 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:52.342381 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:35:52.342473 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:35:52.342504 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:52.342546 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:52.342584 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:35:52.342646 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:35:52.347880 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40543
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:52.348981 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:52.350095 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:52.358593 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:52.358593 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:52.358601 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:52.358844 1475 server_base.cc:1034] running on GCE node
I20260924 04:35:52.359586 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:52.360657 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:52.361860 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790224552361828 us; error 47 us; skew 500 ppm
Sep 24 04:35:52 dist-test-slave-jlnt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:52.364503 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:35:52.365442 1475 webserver.cc:460] Webserver started at http://127.0.98.62:40969/ using document root <none> and password file <none>
I20260924 04:35:52.366071 1475 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:52.366163 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:52.366396 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:52.368422 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "5e23998c5a9c447cac2b2fa9a9293e29"
format_stamp: "Formatted at 2026-09-24 04:35:52 on dist-test-slave-jlnt"
server_key: "742450e8f1e4e17a60783305faa3bf07"
server_key_iv: "296b1c96bbf9d8b41e02f28515b310bc"
server_key_version: "encryptionkey@0"
I20260924 04:35:52.369001 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e23998c5a9c447cac2b2fa9a9293e29"
format_stamp: "Formatted at 2026-09-24 04:35:52 on dist-test-slave-jlnt"
server_key: "742450e8f1e4e17a60783305faa3bf07"
server_key_iv: "296b1c96bbf9d8b41e02f28515b310bc"
server_key_version: "encryptionkey@0"
I20260924 04:35:52.372813 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:35:52.375274 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:52.376601 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:35:52.376762 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "5e23998c5a9c447cac2b2fa9a9293e29"
format_stamp: "Formatted at 2026-09-24 04:35:52 on dist-test-slave-jlnt"
server_key: "742450e8f1e4e17a60783305faa3bf07"
server_key_iv: "296b1c96bbf9d8b41e02f28515b310bc"
server_key_version: "encryptionkey@0"
I20260924 04:35:52.376881 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:52.389757 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:52.392809 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:52.393011 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:52.400671 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40543
I20260924 04:35:52.400691 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40543 every 8 connection(s)
I20260924 04:35:52.401639 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:35:52.404248 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:52.406544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 1475
I20260924 04:35:52.406627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:35:52.406888 392 external_mini_cluster.cc:1428] Setting key 5e0e7ac2dbcecb504a52192fd089952d
I20260924 04:35:52.410346 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29: Bootstrap starting.
I20260924 04:35:52.412880 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:52.413820 1553 log.cc:824] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:52.415723 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29: No bootstrap required, opened a new log
Sep 24 04:35:52 dist-test-slave-jlnt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:52.419286 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e23998c5a9c447cac2b2fa9a9293e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260924 04:35:52.419504 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:52.419586 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e23998c5a9c447cac2b2fa9a9293e29, State: Initialized, Role: FOLLOWER
I20260924 04:35:52.420055 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e23998c5a9c447cac2b2fa9a9293e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260924 04:35:52.420177 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:35:52.420277 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:35:52.420388 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:52.421592 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e23998c5a9c447cac2b2fa9a9293e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260924 04:35:52.421972 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e23998c5a9c447cac2b2fa9a9293e29; no voters:
I20260924 04:35:52.422313 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:35:52.422382 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:52.422605 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [term 1 LEADER]: Becoming Leader. State: Replica: 5e23998c5a9c447cac2b2fa9a9293e29, State: Running, Role: LEADER
I20260924 04:35:52.422698 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:52.408291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.0.98.62:40543)
0924 04:35:52.408701 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:52.408719 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:52.408760 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:35:52.409632 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:52.409654 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:52.410052 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:52.410400 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:52.411104 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.412557 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.413663 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.414367 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.414549 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:52.417462 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:52.417475 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:52.417489 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:52.417516 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:52.419302 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.419828 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.419834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.419846 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.419954 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.420283 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.420291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.420292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.420810 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:52.421038 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:52.421379 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 04:35:52.421716 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260924 04:35:52.422907 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e23998c5a9c447cac2b2fa9a9293e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260924 04:35:52.423597 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:35:52.424057 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e23998c5a9c447cac2b2fa9a9293e29. Latest consensus state: current_term: 1 leader_uuid: "5e23998c5a9c447cac2b2fa9a9293e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e23998c5a9c447cac2b2fa9a9293e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } } }
I20260924 04:35:52.424172 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [sys.catalog]: This master's current role is: LEADER
I20260924 04:35:52.424468 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e23998c5a9c447cac2b2fa9a9293e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e23998c5a9c447cac2b2fa9a9293e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } } }
I20260924 04:35:52.424587 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29 [sys.catalog]: This master's current role is: LEADER
I20260924 04:35:52.425431 1562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:35:52.428531 1562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:35:52.434314 1562 catalog_manager.cc:1347] Generated new cluster ID: 64751236661945448eb5cd32d68399c6
I20260924 04:35:52.434396 1562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:35:52.445871 1562 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:35:52.446828 1562 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:35:52.452615 1562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29: Generated new TSK 0
I20260924 04:35:52.453328 1562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:35:52.484917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:52.597555 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:52.597815 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:52.597891 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:52.601811 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:52.601861 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:52.601897 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:35:52.601987 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:35:52.607048 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:52.608120 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:52.609125 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:35:52.618317 1579 server_base.cc:1034] running on GCE node
W20260924 04:35:52.618253 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:52.618275 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:52.618268 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:52.618837 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:52.619477 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:52.620638 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790224552620613 us; error 39 us; skew 500 ppm
Sep 24 04:35:52 dist-test-slave-jlnt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:52.623589 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:35:52.624578 1579 webserver.cc:460] Webserver started at http://127.0.98.1:44301/ using document root <none> and password file <none>
I20260924 04:35:52.625293 1579 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:52.625360 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:52.625583 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:52.627636 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "82fa0b767d81497389abab6f3609b6d4"
format_stamp: "Formatted at 2026-09-24 04:35:52 on dist-test-slave-jlnt"
server_key: "5254599c9b4c41d2a289945cb101bad1"
server_key_iv: "d032e993695b207db1559eb9e24ea93e"
server_key_version: "encryptionkey@0"
I20260924 04:35:52.628216 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "82fa0b767d81497389abab6f3609b6d4"
format_stamp: "Formatted at 2026-09-24 04:35:52 on dist-test-slave-jlnt"
server_key: "5254599c9b4c41d2a289945cb101bad1"
server_key_iv: "d032e993695b207db1559eb9e24ea93e"
server_key_version: "encryptionkey@0"
I20260924 04:35:52.631929 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:35:52.634294 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:52.635447 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:35:52.635573 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "82fa0b767d81497389abab6f3609b6d4"
format_stamp: "Formatted at 2026-09-24 04:35:52 on dist-test-slave-jlnt"
server_key: "5254599c9b4c41d2a289945cb101bad1"
server_key_iv: "d032e993695b207db1559eb9e24ea93e"
server_key_version: "encryptionkey@0"
I20260924 04:35:52.635682 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:52.653744 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:52.657073 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:52.657307 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:52.658834 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:52.658913 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:52.659001 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:52.659051 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:35:52 dist-test-slave-jlnt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:52.669972 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46373
I20260924 04:35:52.670094 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46373 every 8 connection(s)
I20260924 04:35:52.671141 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:35:52.673848 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:52.659676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53921 (local address 127.0.98.62:40543)
0924 04:35:52.659790 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:52.659794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:52.660951 (+ 1157us) server_negotiation.cc:407] Connection header received
0924 04:35:52.662126 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:52.662132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:52.662202 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:52.662284 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:52.664183 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.665105 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.666718 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.667068 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.667194 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:52.670173 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:52.670176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:52.670179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:52.670206 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:52.672005 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.672519 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.672521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.672523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.672605 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.672867 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.672869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.672870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.673036 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:52.673115 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:52.673624 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 04:35:52.673731 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:35:52.674644 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:52.660209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.1:53921)
0924 04:35:52.660789 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:52.660834 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:52.661801 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:52.662804 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:52.662813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:52.663261 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:52.664016 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:52.664036 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.665255 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.665260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:52.666586 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:52.666596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.667232 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.667237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:52.667423 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:52.668139 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:52.668156 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:52.669944 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:52.672121 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:52.672126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:52.672140 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:52.672420 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:52.672667 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:52.672669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:52.672671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:52.672786 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:52.673128 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:52.673132 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:52.673393 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 04:35:52.673546 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 04:35:52.673813 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":141,"threads_started":1}
I20260924 04:35:52.675832 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40543
I20260924 04:35:52.675961 1710 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:52.676646 1710 heartbeater.cc:511] Master 127.0.98.62:40543 requested a full tablet report, sending...
I20260924 04:35:52.678397 1506 ts_manager.cc:194] Registered new tserver with Master: 82fa0b767d81497389abab6f3609b6d4 (127.0.98.1:46373)
I20260924 04:35:52.679491 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53921
I20260924 04:35:52.681128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 1579
I20260924 04:35:52.681281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:35:52.681604 392 external_mini_cluster.cc:1428] Setting key 787e73b6b1666bf888a3be769b2b90fb
I20260924 04:35:52.687004 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:52.678200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43145 (local address 127.0.98.62:40543)
0924 04:35:52.678308 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:52.678311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:52.680149 (+ 1838us) server_negotiation.cc:407] Connection header received
0924 04:35:52.680295 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:52.680300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:52.680369 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:52.680447 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:52.680887 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.681668 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.682458 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.682693 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.682800 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:52.683915 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:52.683920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:52.683927 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:52.683947 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:52.685418 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.685866 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.685871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.685872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.685947 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.686232 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.686239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.686240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.686404 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:52.686509 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:52.686751 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:35:52.686853 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:35:52.687017 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:52.678107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.1:43145)
0924 04:35:52.680076 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:52.680100 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:52.680190 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:52.680481 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:52.680484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:52.680672 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:52.680761 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:52.680766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.681777 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.681781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:52.682337 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:52.682350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.682802 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.682805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:52.682894 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:52.683417 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:52.683428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:52.683779 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:52.685559 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:52.685563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:52.685565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:52.685768 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:52.686042 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:52.686045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:52.686047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:52.686138 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:52.686498 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:52.686501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:52.686590 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:35:52.686702 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 04:35:52.686841 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1862,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:35:52.709797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:52.819554 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:52.819770 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:52.819811 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:52.823726 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:52.823772 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:52.823808 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:35:52.823839 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:35:52.829041 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:52.829978 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:52.830888 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:52.840301 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:52.840291 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:52.840286 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:52.840497 1719 server_base.cc:1034] running on GCE node
I20260924 04:35:52.840998 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:52.841657 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:52.842866 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790224552842844 us; error 27 us; skew 500 ppm
Sep 24 04:35:52 dist-test-slave-jlnt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:52.845781 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:35:52.846675 1719 webserver.cc:460] Webserver started at http://127.0.98.2:42297/ using document root <none> and password file <none>
I20260924 04:35:52.847230 1719 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:52.847319 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:52.847558 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:52.849645 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "47271a5d3c47472480fd400480111974"
format_stamp: "Formatted at 2026-09-24 04:35:52 on dist-test-slave-jlnt"
server_key: "fa12a593134a5fc223402557a6fc7860"
server_key_iv: "110f66ab0892812bb770b7b1e8b3bce0"
server_key_version: "encryptionkey@0"
I20260924 04:35:52.850194 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "47271a5d3c47472480fd400480111974"
format_stamp: "Formatted at 2026-09-24 04:35:52 on dist-test-slave-jlnt"
server_key: "fa12a593134a5fc223402557a6fc7860"
server_key_iv: "110f66ab0892812bb770b7b1e8b3bce0"
server_key_version: "encryptionkey@0"
I20260924 04:35:52.853891 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:35:52.856508 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:52.857834 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:35:52.857987 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "47271a5d3c47472480fd400480111974"
format_stamp: "Formatted at 2026-09-24 04:35:52 on dist-test-slave-jlnt"
server_key: "fa12a593134a5fc223402557a6fc7860"
server_key_iv: "110f66ab0892812bb770b7b1e8b3bce0"
server_key_version: "encryptionkey@0"
I20260924 04:35:52.858108 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:52.876170 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:52.879132 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:52.879343 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:52.880882 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:52.880962 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:52.881037 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:52.881075 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:35:52 dist-test-slave-jlnt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:52.893007 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39807
I20260924 04:35:52.893096 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39807 every 8 connection(s)
I20260924 04:35:52.893965 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:35:52.895622 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:52.881820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49997 (local address 127.0.98.62:40543)
0924 04:35:52.882003 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:52.882011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:52.882796 (+ 785us) server_negotiation.cc:407] Connection header received
0924 04:35:52.883787 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:52.883794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:52.883860 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:52.883971 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:52.885617 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.886222 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.887376 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.887592 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.887713 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:52.891673 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:52.891676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:52.891679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:52.891712 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:52.893395 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.893903 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.893907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.893909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.893994 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.894362 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.894368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.894373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.894607 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:52.894717 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:52.895296 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 04:35:52.895420 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:35:52.895817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 1719
I20260924 04:35:52.895907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:35:52.896214 392 external_mini_cluster.cc:1428] Setting key d0388fb9396075e8096a0f7d8cd6524a
I20260924 04:35:52.896486 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:52.882200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.2:49997)
0924 04:35:52.882632 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:52.882680 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:52.883543 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:52.884229 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:52.884241 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:52.884779 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:52.885424 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:52.885439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.886354 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.886357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:52.887242 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:52.887251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.887749 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.887753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:52.887928 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:52.888667 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:52.888693 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:52.891494 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:52.893507 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:52.893514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:52.893528 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:52.893790 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:52.894117 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:52.894122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:52.894129 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:52.894260 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:52.894778 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:52.894784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:52.895064 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 04:35:52.895253 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 04:35:52.895566 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":131,"threads_started":1}
I20260924 04:35:52.897742 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40543
I20260924 04:35:52.897886 1850 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:52.898442 1850 heartbeater.cc:511] Master 127.0.98.62:40543 requested a full tablet report, sending...
I20260924 04:35:52.899554 1506 ts_manager.cc:194] Registered new tserver with Master: 47271a5d3c47472480fd400480111974 (127.0.98.2:39807)
I20260924 04:35:52.900167 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49997
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:35:52.909551 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:52.900637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.2:37609)
0924 04:35:52.902552 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:52.902569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:52.902679 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:52.902975 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:52.902977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:52.903156 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:52.903245 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:52.903250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.904371 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.904374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:52.904920 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:52.904927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.905558 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.905560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:52.905631 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:52.906197 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:52.906208 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:52.906504 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:52.908068 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:52.908071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:52.908073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:52.908265 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:52.908501 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:52.908504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:52.908506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:52.908592 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:52.908963 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:52.908968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:52.909091 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 04:35:52.909231 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 04:35:52.909361 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1819,"thread_start_us":101,"threads_started":1}
I20260924 04:35:52.909580 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:52.900754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37609 (local address 127.0.98.62:40543)
0924 04:35:52.900877 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:52.900882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:52.902672 (+ 1790us) server_negotiation.cc:407] Connection header received
0924 04:35:52.902805 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:52.902810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:52.902864 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:52.902951 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:52.903356 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.904254 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.905037 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:52.905448 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:52.905570 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:52.906643 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:52.906647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:52.906651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:52.906691 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:52.907973 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.908345 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.908348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.908349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.908413 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:52.908679 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:52.908681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:52.908682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:52.908860 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:52.908934 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:52.909287 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 04:35:52.909412 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:35:52.924976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:53.034346 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:53.034617 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:53.034698 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:53.039342 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:53.039402 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:53.039477 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:35:53.039558 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:35:53.044847 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:53.045981 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:53.047011 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:53.056559 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:53.056636 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:53.056725 1859 server_base.cc:1034] running on GCE node
W20260924 04:35:53.056593 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:53.057226 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:53.057858 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:53.059029 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790224553058986 us; error 47 us; skew 500 ppm
Sep 24 04:35:53 dist-test-slave-jlnt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:53.061925 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:35:53.062922 1859 webserver.cc:460] Webserver started at http://127.0.98.3:42869/ using document root <none> and password file <none>
I20260924 04:35:53.063465 1859 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:53.063555 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:53.063782 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:53.065842 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e71f16f0f3f4ac0951e9639b46b5d57"
format_stamp: "Formatted at 2026-09-24 04:35:53 on dist-test-slave-jlnt"
server_key: "d9bcb0524c4acb91f1845cd0f74e5a4a"
server_key_iv: "bcdef328fa38a7b36af2d809c4076b58"
server_key_version: "encryptionkey@0"
I20260924 04:35:53.066402 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e71f16f0f3f4ac0951e9639b46b5d57"
format_stamp: "Formatted at 2026-09-24 04:35:53 on dist-test-slave-jlnt"
server_key: "d9bcb0524c4acb91f1845cd0f74e5a4a"
server_key_iv: "bcdef328fa38a7b36af2d809c4076b58"
server_key_version: "encryptionkey@0"
I20260924 04:35:53.070115 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:35:53.072528 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:53.073770 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:35:53.073897 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "2e71f16f0f3f4ac0951e9639b46b5d57"
format_stamp: "Formatted at 2026-09-24 04:35:53 on dist-test-slave-jlnt"
server_key: "d9bcb0524c4acb91f1845cd0f74e5a4a"
server_key_iv: "bcdef328fa38a7b36af2d809c4076b58"
server_key_version: "encryptionkey@0"
I20260924 04:35:53.074005 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:53.096186 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:53.099215 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:53.099418 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:53.101089 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:53.101181 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:53.101250 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:53.101281 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:35:53 dist-test-slave-jlnt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:53.112306 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40817
I20260924 04:35:53.112375 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40817 every 8 connection(s)
I20260924 04:35:53.113420 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:35:53.115353 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.102030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49131 (local address 127.0.98.62:40543)
0924 04:35:53.102197 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.102203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.103223 (+ 1020us) server_negotiation.cc:407] Connection header received
0924 04:35:53.104566 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.104570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.104630 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.104754 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:53.105909 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.106441 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.108101 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.108508 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.108638 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.111734 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:53.111737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:53.111740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:53.111763 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:53.113452 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.113978 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.113982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.113984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.114075 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.114448 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.114451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.114452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.114650 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:53.114731 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.115141 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.115241 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:35:53.116132 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.102508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.3:49131)
0924 04:35:53.103024 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.103093 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.104268 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.104919 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.104928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.105327 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:53.105771 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.105784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.106575 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.106578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.107911 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.107925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.108682 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.108687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.108885 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.109513 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:53.109539 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:53.111554 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:53.113564 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:53.113570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:53.113584 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:53.113858 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:53.114185 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:53.114188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:53.114190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:53.114338 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:53.114738 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:53.114742 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:53.114940 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.115080 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.115333 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":142,"threads_started":1}
I20260924 04:35:53.117391 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40543
I20260924 04:35:53.117527 1990 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:53.118049 1990 heartbeater.cc:511] Master 127.0.98.62:40543 requested a full tablet report, sending...
I20260924 04:35:53.119140 1505 ts_manager.cc:194] Registered new tserver with Master: 2e71f16f0f3f4ac0951e9639b46b5d57 (127.0.98.3:40817)
I20260924 04:35:53.119927 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49131
I20260924 04:35:53.121022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 1859
I20260924 04:35:53.121184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:35:53.121461 392 external_mini_cluster.cc:1428] Setting key f3969a786660e1bbdbae76fadd647060
I20260924 04:35:53.123401 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:35:53.127118 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.119499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36451 (local address 127.0.98.62:40543)
0924 04:35:53.120215 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.120218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.120228 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:35:53.120253 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.120254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.120281 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.120368 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:53.120774 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.121394 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.122394 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.122714 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.122809 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.123910 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:53.123914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:53.123916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:53.123939 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:53.125601 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.126002 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.126007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.126009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.126097 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.126390 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.126393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.126394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.126553 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:53.126631 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.126928 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.127013 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663}
I20260924 04:35:53.127331 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.119392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.3:36451)
0924 04:35:53.119674 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.119688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.119764 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.120383 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.120386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.120570 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:53.120665 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.120670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.121512 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.121519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.122264 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.122275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.122817 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.122820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.122891 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.123411 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:53.123425 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:53.123784 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:53.125705 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:53.125709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:53.125711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:53.125923 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:53.126195 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:53.126200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:53.126205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:53.126299 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:53.126648 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:53.126654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:53.126767 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.126882 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.127037 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
Sep 24 04:35:53 dist-test-slave-jlnt krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224553, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:35:53 dist-test-slave-jlnt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224553, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:53.160058 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.150562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.0.98.62:40543)
0924 04:35:53.150790 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.150794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.150810 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:35:53.150860 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.150863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.150918 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.151024 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:53.152126 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.152688 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.153882 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.154122 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.154233 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.156956 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:53.156960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:53.156962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:53.156988 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:53.158364 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.158781 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.158786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.158788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.158868 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.159152 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.159155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.159157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.159418 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:53.159558 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.159787 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.159897 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 04:35:53.163004 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:35:53.165997 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:35:53.181998 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.175809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46373 (local address 127.0.0.1:40752)
0924 04:35:53.176342 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.176393 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.176516 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.177305 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.177310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.177334 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:53.177486 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.177493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.178997 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.179001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.180280 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.180291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.181470 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.181473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.181587 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.181640 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.181738 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.181851 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260924 04:35:53.182349 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.176295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39807 (local address 127.0.0.1:50758)
0924 04:35:53.176812 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.176824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.176924 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.177672 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.177675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.177687 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:53.177898 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.177904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.179720 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.179725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.180838 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.180846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.181781 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.181786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.181867 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.181888 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.181953 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.182028 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":170,"thread_start_us":83,"threads_started":1}
I20260924 04:35:53.182729 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.175934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.0.98.1:46373)
0924 04:35:53.176786 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.176793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.176816 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:35:53.176895 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.176902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.177114 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.177648 (+ 534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:53.177789 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.178870 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.180460 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.181331 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.182295 (+ 964us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.182435 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.182486 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.182570 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":71,"threads_started":1}
I20260924 04:35:53.182951 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.176933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.0.98.2:39807)
0924 04:35:53.177267 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.177278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.177297 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:35:53.177359 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.177363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.177504 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.177762 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:53.177986 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.179571 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.180958 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.181642 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.181761 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.182631 (+ 870us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.182686 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.182777 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260924 04:35:53.183099 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.176010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40817 (local address 127.0.0.1:52804)
0924 04:35:53.176753 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.176766 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.176844 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.177617 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.177622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.177643 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:53.177753 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.177759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.180329 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.180335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.181531 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.181538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.182757 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.182762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.182859 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.182882 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.182929 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.182986 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":7,"thread_start_us":115,"threads_started":1}
I20260924 04:35:53.183791 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.176570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.0.98.3:40817)
0924 04:35:53.177099 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.177112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.177134 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:35:53.177240 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.177246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.177445 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.178050 (+ 605us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:53.178174 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.180197 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.181721 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.182621 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.183351 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.183500 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.183550 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.183632 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":109,"threads_started":1}
I20260924 04:35:53.185505 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 29507546e13448fbb5f936cc3c4db66e (DEFAULT_TABLE table=test-table [id=c416ceed58024f5bb2d4356db8ca19a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:53.185565 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 29507546e13448fbb5f936cc3c4db66e (DEFAULT_TABLE table=test-table [id=c416ceed58024f5bb2d4356db8ca19a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:53.186172 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 29507546e13448fbb5f936cc3c4db66e (DEFAULT_TABLE table=test-table [id=c416ceed58024f5bb2d4356db8ca19a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:53.186565 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29507546e13448fbb5f936cc3c4db66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:53.186566 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29507546e13448fbb5f936cc3c4db66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:53.186969 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29507546e13448fbb5f936cc3c4db66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:53.192266 2013 tablet_bootstrap.cc:492] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: Bootstrap starting.
I20260924 04:35:53.193871 2014 tablet_bootstrap.cc:492] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: Bootstrap starting.
I20260924 04:35:53.194249 2015 tablet_bootstrap.cc:492] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: Bootstrap starting.
I20260924 04:35:53.194811 2013 tablet_bootstrap.cc:654] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:53.195662 2013 log.cc:824] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:53.196264 2015 tablet_bootstrap.cc:654] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:53.197078 2015 log.cc:824] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:53.197580 2013 tablet_bootstrap.cc:492] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: No bootstrap required, opened a new log
I20260924 04:35:53.197780 2013 ts_tablet_manager.cc:1397] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:35:53.198403 2014 tablet_bootstrap.cc:654] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:53.199013 2015 tablet_bootstrap.cc:492] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: No bootstrap required, opened a new log
I20260924 04:35:53.199185 2015 ts_tablet_manager.cc:1397] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:35:53.199155 2014 log.cc:824] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:53.200910 2014 tablet_bootstrap.cc:492] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: No bootstrap required, opened a new log
I20260924 04:35:53.201094 2014 ts_tablet_manager.cc:1397] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 04:35:53.201335 2013 raft_consensus.cc:348] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } }
I20260924 04:35:53.201558 2013 raft_consensus.cc:374] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:53.201627 2013 raft_consensus.cc:729] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82fa0b767d81497389abab6f3609b6d4, State: Initialized, Role: FOLLOWER
I20260924 04:35:53.202045 2013 consensus_queue.cc:260] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } }
I20260924 04:35:53.202600 1710 heartbeater.cc:503] Master 127.0.98.62:40543 was elected leader, sending a full tablet report...
I20260924 04:35:53.202944 2013 ts_tablet_manager.cc:1428] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:35:53.202836 2015 raft_consensus.cc:348] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } }
I20260924 04:35:53.203058 2015 raft_consensus.cc:374] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:53.203106 2015 raft_consensus.cc:729] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47271a5d3c47472480fd400480111974, State: Initialized, Role: FOLLOWER
I20260924 04:35:53.203496 2015 consensus_queue.cc:260] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } }
I20260924 04:35:53.204396 2015 ts_tablet_manager.cc:1428] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:35:53.204766 1850 heartbeater.cc:503] Master 127.0.98.62:40543 was elected leader, sending a full tablet report...
I20260924 04:35:53.204780 2014 raft_consensus.cc:348] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } }
I20260924 04:35:53.204974 2014 raft_consensus.cc:374] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:53.205005 2014 raft_consensus.cc:729] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e71f16f0f3f4ac0951e9639b46b5d57, State: Initialized, Role: FOLLOWER
I20260924 04:35:53.205384 2014 consensus_queue.cc:260] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } }
I20260924 04:35:53.206285 2014 ts_tablet_manager.cc:1428] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260924 04:35:53.206616 1990 heartbeater.cc:503] Master 127.0.98.62:40543 was elected leader, sending a full tablet report...
W20260924 04:35:53.364398 1991 tablet.cc:2369] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:35:53.395273 1851 tablet.cc:2369] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:35:53.395272 2020 raft_consensus.cc:482] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:35:53.395609 2020 raft_consensus.cc:504] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } }
I20260924 04:35:53.396839 2020 leader_election.cc:288] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82fa0b767d81497389abab6f3609b6d4 (127.0.98.1:46373), 2e71f16f0f3f4ac0951e9639b46b5d57 (127.0.98.3:40817)
I20260924 04:35:53.400947 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.397269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42589 (local address 127.0.98.3:40817)
0924 04:35:53.397424 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.397430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.397678 (+ 248us) server_negotiation.cc:407] Connection header received
0924 04:35:53.397851 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.397853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.397923 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.397999 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:53.398272 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.398920 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.399927 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.400359 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.400478 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.400550 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.400770 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.400835 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 04:35:53.400983 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.397122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40817 (local address 127.0.98.2:42589)
0924 04:35:53.397564 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.397575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.397734 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.398041 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.398043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.398058 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:53.398140 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.398144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.399030 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.399032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.399812 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.399819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.400485 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.400489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.400592 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.400666 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.400741 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.400823 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":48,"thread_start_us":76,"threads_started":1}
I20260924 04:35:53.401067 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.397260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57645 (local address 127.0.98.1:46373)
0924 04:35:53.397408 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.397415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.397531 (+ 116us) server_negotiation.cc:407] Connection header received
0924 04:35:53.397775 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.397780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.397848 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.397906 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:53.398228 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.398840 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.399959 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.400372 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.400486 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.400549 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.400871 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.400941 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:35:53.401443 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.397077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46373 (local address 127.0.98.2:57645)
0924 04:35:53.397423 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.397438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.397587 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.397941 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.397946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.397969 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:53.398111 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.398117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.398938 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.398944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.399814 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.399823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.400599 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.400602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.400747 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.400764 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.401257 (+ 493us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.401319 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":10,"thread_start_us":89,"threads_started":1}
I20260924 04:35:53.401746 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29507546e13448fbb5f936cc3c4db66e" candidate_uuid: "47271a5d3c47472480fd400480111974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" is_pre_election: true
I20260924 04:35:53.402065 1935 raft_consensus.cc:2454] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47271a5d3c47472480fd400480111974 in term 0.
I20260924 04:35:53.402091 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29507546e13448fbb5f936cc3c4db66e" candidate_uuid: "47271a5d3c47472480fd400480111974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82fa0b767d81497389abab6f3609b6d4" is_pre_election: true
I20260924 04:35:53.402362 1664 raft_consensus.cc:2454] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47271a5d3c47472480fd400480111974 in term 0.
I20260924 04:35:53.402570 1735 leader_election.cc:302] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e71f16f0f3f4ac0951e9639b46b5d57, 47271a5d3c47472480fd400480111974; no voters:
I20260924 04:35:53.402822 2020 raft_consensus.cc:2781] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:35:53.402899 2020 raft_consensus.cc:482] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:35:53.402940 2020 raft_consensus.cc:3037] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:53.404224 2020 raft_consensus.cc:504] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } }
I20260924 04:35:53.404587 2020 leader_election.cc:288] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [CANDIDATE]: Term 1 election: Requested vote from peers 82fa0b767d81497389abab6f3609b6d4 (127.0.98.1:46373), 2e71f16f0f3f4ac0951e9639b46b5d57 (127.0.98.3:40817)
I20260924 04:35:53.404846 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29507546e13448fbb5f936cc3c4db66e" candidate_uuid: "47271a5d3c47472480fd400480111974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82fa0b767d81497389abab6f3609b6d4"
I20260924 04:35:53.404917 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29507546e13448fbb5f936cc3c4db66e" candidate_uuid: "47271a5d3c47472480fd400480111974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57"
I20260924 04:35:53.404984 1664 raft_consensus.cc:3037] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:53.405063 1935 raft_consensus.cc:3037] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:53.406067 1664 raft_consensus.cc:2454] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47271a5d3c47472480fd400480111974 in term 1.
I20260924 04:35:53.406411 1736 leader_election.cc:302] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47271a5d3c47472480fd400480111974, 82fa0b767d81497389abab6f3609b6d4; no voters:
I20260924 04:35:53.406616 2020 raft_consensus.cc:2781] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:53.406780 1935 raft_consensus.cc:2454] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47271a5d3c47472480fd400480111974 in term 1.
I20260924 04:35:53.406862 2020 raft_consensus.cc:686] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 1 LEADER]: Becoming Leader. State: Replica: 47271a5d3c47472480fd400480111974, State: Running, Role: LEADER
I20260924 04:35:53.407631 2020 consensus_queue.cc:237] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } }
I20260924 04:35:53.410504 1505 catalog_manager.cc:5495] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47271a5d3c47472480fd400480111974 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "47271a5d3c47472480fd400480111974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } health_report { overall_health: HEALTHY } } }
W20260924 04:35:53.422780 1711 tablet.cc:2369] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:35:53.477615 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.473793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.0.98.2:39807)
0924 04:35:53.473916 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.473920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.473934 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:35:53.474138 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.474140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.474198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.474285 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:53.474520 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.475035 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.476015 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.476252 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.476402 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.476691 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:53.477104 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:53.477329 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.477380 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.477441 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:35:53.483624 1664 raft_consensus.cc:1261] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Refusing update from remote peer 47271a5d3c47472480fd400480111974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:53.483639 1935 raft_consensus.cc:1261] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Refusing update from remote peer 47271a5d3c47472480fd400480111974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:53.484308 2024 consensus_queue.cc:1035] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:53.484494 2020 consensus_queue.cc:1035] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:53.494261 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332759771062026240 to 7332759770760417280. Current Snapshot: MvccSnapshot[applied={T|T < 7332759771062026240}]
I20260924 04:35:53.502265 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:53.502439 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:53.504668 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29: Sending DeleteTablet for 3 replicas of tablet 29507546e13448fbb5f936cc3c4db66e
I20260924 04:35:53.505494 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 29507546e13448fbb5f936cc3c4db66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52804
I20260924 04:35:53.505538 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 29507546e13448fbb5f936cc3c4db66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50758
I20260924 04:35:53.505975 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 29507546e13448fbb5f936cc3c4db66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40752
I20260924 04:35:53.506134 2037 tablet_replica.cc:330] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: stopping tablet replica
I20260924 04:35:53.506171 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:35:53.506417 2037 raft_consensus.cc:2229] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 04:35:53.506525 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:35:53.506649 2038 tablet_replica.cc:330] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: stopping tablet replica
I20260924 04:35:53.506714 2037 raft_consensus.cc:2258] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:53.506951 2038 raft_consensus.cc:2229] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:35:53.507305 2038 raft_consensus.cc:2258] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:53.508143 2039 tablet_replica.cc:330] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: stopping tablet replica
I20260924 04:35:53.508232 2037 ts_tablet_manager.cc:1905] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:53.508463 2039 raft_consensus.cc:2229] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:35:53.508795 2039 raft_consensus.cc:2258] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:53.511509 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 52c0345f598a48debc6f26dad8b05655 (DEFAULT_TABLE table=test-table [id=e64cdc9a2d8e4b8db65bb36670981161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:53.511759 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c0345f598a48debc6f26dad8b05655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:53.511804 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 52c0345f598a48debc6f26dad8b05655 (DEFAULT_TABLE table=test-table [id=e64cdc9a2d8e4b8db65bb36670981161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:53.512027 2039 ts_tablet_manager.cc:1905] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:53.513238 2038 ts_tablet_manager.cc:1905] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:53.514313 2039 ts_tablet_manager.cc:1918] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:35:53.514407 2039 log.cc:1196] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/29507546e13448fbb5f936cc3c4db66e
I20260924 04:35:53.514477 2013 tablet_bootstrap.cc:492] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: Bootstrap starting.
I20260924 04:35:53.514752 2039 ts_tablet_manager.cc:1939] T 29507546e13448fbb5f936cc3c4db66e P 82fa0b767d81497389abab6f3609b6d4: Deleting consensus metadata
I20260924 04:35:53.515448 2013 tablet_bootstrap.cc:654] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:53.516183 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 52c0345f598a48debc6f26dad8b05655 (DEFAULT_TABLE table=test-table [id=e64cdc9a2d8e4b8db65bb36670981161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:53.516388 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c0345f598a48debc6f26dad8b05655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:53.516530 2038 ts_tablet_manager.cc:1918] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:35:53.516620 2038 log.cc:1196] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/29507546e13448fbb5f936cc3c4db66e
I20260924 04:35:53.517102 2038 ts_tablet_manager.cc:1939] T 29507546e13448fbb5f936cc3c4db66e P 47271a5d3c47472480fd400480111974: Deleting consensus metadata
I20260924 04:35:53.518198 1494 catalog_manager.cc:4841] TS 47271a5d3c47472480fd400480111974 (127.0.98.2:39807): tablet 29507546e13448fbb5f936cc3c4db66e (table test-table [id=c416ceed58024f5bb2d4356db8ca19a2]) successfully deleted
I20260924 04:35:53.518589 2015 tablet_bootstrap.cc:492] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: Bootstrap starting.
I20260924 04:35:53.518970 1492 catalog_manager.cc:4841] TS 82fa0b767d81497389abab6f3609b6d4 (127.0.98.1:46373): tablet 29507546e13448fbb5f936cc3c4db66e (table test-table [id=c416ceed58024f5bb2d4356db8ca19a2]) successfully deleted
I20260924 04:35:53.519412 2013 tablet_bootstrap.cc:492] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: No bootstrap required, opened a new log
I20260924 04:35:53.519481 2015 tablet_bootstrap.cc:654] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:53.519520 2013 ts_tablet_manager.cc:1397] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 04:35:53.520043 2013 raft_consensus.cc:348] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.520174 2013 raft_consensus.cc:374] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:53.520215 2013 raft_consensus.cc:729] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82fa0b767d81497389abab6f3609b6d4, State: Initialized, Role: FOLLOWER
I20260924 04:35:53.520345 2013 consensus_queue.cc:260] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.520745 2015 tablet_bootstrap.cc:492] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: No bootstrap required, opened a new log
I20260924 04:35:53.520825 2015 ts_tablet_manager.cc:1397] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:35:53.521309 2013 ts_tablet_manager.cc:1428] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:35:53.521293 2015 raft_consensus.cc:348] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.521411 2015 raft_consensus.cc:374] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:53.521449 2015 raft_consensus.cc:729] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47271a5d3c47472480fd400480111974, State: Initialized, Role: FOLLOWER
I20260924 04:35:53.521585 2015 consensus_queue.cc:260] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.521723 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c0345f598a48debc6f26dad8b05655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:53.521837 2015 ts_tablet_manager.cc:1428] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:53.523733 2014 tablet_bootstrap.cc:492] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: Bootstrap starting.
I20260924 04:35:53.524700 2014 tablet_bootstrap.cc:654] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:53.526185 2037 ts_tablet_manager.cc:1918] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:35:53.526333 2037 log.cc:1196] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/29507546e13448fbb5f936cc3c4db66e
I20260924 04:35:53.526435 2014 tablet_bootstrap.cc:492] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: No bootstrap required, opened a new log
I20260924 04:35:53.526525 2014 ts_tablet_manager.cc:1397] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 04:35:53.526736 2037 ts_tablet_manager.cc:1939] T 29507546e13448fbb5f936cc3c4db66e P 2e71f16f0f3f4ac0951e9639b46b5d57: Deleting consensus metadata
I20260924 04:35:53.526944 2014 raft_consensus.cc:348] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.527055 2014 raft_consensus.cc:374] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:53.527086 2014 raft_consensus.cc:729] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e71f16f0f3f4ac0951e9639b46b5d57, State: Initialized, Role: FOLLOWER
I20260924 04:35:53.527204 2014 consensus_queue.cc:260] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.527454 2014 ts_tablet_manager.cc:1428] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:35:53.528326 1491 catalog_manager.cc:4841] TS 2e71f16f0f3f4ac0951e9639b46b5d57 (127.0.98.3:40817): tablet 29507546e13448fbb5f936cc3c4db66e (table test-table [id=c416ceed58024f5bb2d4356db8ca19a2]) successfully deleted
I20260924 04:35:53.622438 2034 raft_consensus.cc:482] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:35:53.622571 2034 raft_consensus.cc:504] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.623092 2034 leader_election.cc:288] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82fa0b767d81497389abab6f3609b6d4 (127.0.98.1:46373), 2e71f16f0f3f4ac0951e9639b46b5d57 (127.0.98.3:40817)
I20260924 04:35:53.623533 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c0345f598a48debc6f26dad8b05655" candidate_uuid: "47271a5d3c47472480fd400480111974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82fa0b767d81497389abab6f3609b6d4" is_pre_election: true
I20260924 04:35:53.623564 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c0345f598a48debc6f26dad8b05655" candidate_uuid: "47271a5d3c47472480fd400480111974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" is_pre_election: true
I20260924 04:35:53.623692 1664 raft_consensus.cc:2454] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47271a5d3c47472480fd400480111974 in term 0.
I20260924 04:35:53.623692 1935 raft_consensus.cc:2454] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47271a5d3c47472480fd400480111974 in term 0.
I20260924 04:35:53.624013 1736 leader_election.cc:302] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47271a5d3c47472480fd400480111974, 82fa0b767d81497389abab6f3609b6d4; no voters:
I20260924 04:35:53.624200 2034 raft_consensus.cc:2781] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:35:53.624261 2034 raft_consensus.cc:482] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:35:53.624336 2034 raft_consensus.cc:3037] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:53.625528 2034 raft_consensus.cc:504] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.625898 2034 leader_election.cc:288] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [CANDIDATE]: Term 1 election: Requested vote from peers 82fa0b767d81497389abab6f3609b6d4 (127.0.98.1:46373), 2e71f16f0f3f4ac0951e9639b46b5d57 (127.0.98.3:40817)
I20260924 04:35:53.626240 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c0345f598a48debc6f26dad8b05655" candidate_uuid: "47271a5d3c47472480fd400480111974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57"
I20260924 04:35:53.626294 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c0345f598a48debc6f26dad8b05655" candidate_uuid: "47271a5d3c47472480fd400480111974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82fa0b767d81497389abab6f3609b6d4"
I20260924 04:35:53.626369 1935 raft_consensus.cc:3037] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:53.626448 1664 raft_consensus.cc:3037] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:53.627756 1664 raft_consensus.cc:2454] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47271a5d3c47472480fd400480111974 in term 1.
I20260924 04:35:53.627800 1935 raft_consensus.cc:2454] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47271a5d3c47472480fd400480111974 in term 1.
I20260924 04:35:53.628140 1736 leader_election.cc:302] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47271a5d3c47472480fd400480111974, 82fa0b767d81497389abab6f3609b6d4; no voters:
I20260924 04:35:53.628384 2034 raft_consensus.cc:2781] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:53.628481 2034 raft_consensus.cc:686] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 1 LEADER]: Becoming Leader. State: Replica: 47271a5d3c47472480fd400480111974, State: Running, Role: LEADER
I20260924 04:35:53.628623 2034 consensus_queue.cc:237] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.630350 1506 catalog_manager.cc:5495] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47271a5d3c47472480fd400480111974 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "47271a5d3c47472480fd400480111974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:35:53 dist-test-slave-jlnt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:53.678514 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.658832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.0.1:52294)
0924 04:35:53.659121 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.659137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.659253 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.659479 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.659481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.659853 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:53.659947 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.659952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.660746 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.660749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.661370 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.661378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.661822 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.661824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.661883 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.673792 (+ 11909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:53.673804 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:53.675176 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:53.676896 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:53.676908 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:53.676914 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:53.677251 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:53.677528 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:53.677534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:53.677536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:53.677621 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:53.678009 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:53.678015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:53.678124 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.678238 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.678382 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":126,"threads_started":1}
I20260924 04:35:53.678742 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.658865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.0.98.62:40543)
0924 04:35:53.658983 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.658986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.659221 (+ 235us) server_negotiation.cc:407] Connection header received
0924 04:35:53.659329 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.659331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.659369 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.659443 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:53.660043 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.660650 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.661462 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.661727 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.661803 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.675324 (+ 13521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:53.675327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:53.675329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:53.675348 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:53.676773 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.677343 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.677349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.677351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.677427 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.677726 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.677729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.677730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.677909 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:53.678011 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.678357 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.678483 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:35:53.682401 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52294:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:35:53.683156 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:35:53.687599 1784 tablet_service.cc:1467] Processing CreateTablet for tablet c19e3e76f56244de92679ceed9ff645e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2c5dfe9ed9247efa622f0013f0ee07c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:35:53.687599 1644 tablet_service.cc:1467] Processing CreateTablet for tablet c19e3e76f56244de92679ceed9ff645e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2c5dfe9ed9247efa622f0013f0ee07c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:35:53.687835 1924 tablet_service.cc:1467] Processing CreateTablet for tablet c19e3e76f56244de92679ceed9ff645e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2c5dfe9ed9247efa622f0013f0ee07c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:35:53.687968 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19e3e76f56244de92679ceed9ff645e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:53.687968 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19e3e76f56244de92679ceed9ff645e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:53.688199 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19e3e76f56244de92679ceed9ff645e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:53.690618 2014 tablet_bootstrap.cc:492] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: Bootstrap starting.
I20260924 04:35:53.690636 2015 tablet_bootstrap.cc:492] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: Bootstrap starting.
I20260924 04:35:53.691026 2013 tablet_bootstrap.cc:492] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: Bootstrap starting.
I20260924 04:35:53.691465 2014 tablet_bootstrap.cc:654] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:53.691475 2015 tablet_bootstrap.cc:654] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:53.691881 2013 tablet_bootstrap.cc:654] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:53.692703 2015 tablet_bootstrap.cc:492] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: No bootstrap required, opened a new log
I20260924 04:35:53.692792 2015 ts_tablet_manager.cc:1397] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:35:53.692942 2014 tablet_bootstrap.cc:492] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: No bootstrap required, opened a new log
I20260924 04:35:53.693023 2014 ts_tablet_manager.cc:1397] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 04:35:53.693035 2013 tablet_bootstrap.cc:492] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: No bootstrap required, opened a new log
I20260924 04:35:53.693106 2013 ts_tablet_manager.cc:1397] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:35:53.693244 2015 raft_consensus.cc:348] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.693346 2015 raft_consensus.cc:374] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:53.693375 2015 raft_consensus.cc:729] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47271a5d3c47472480fd400480111974, State: Initialized, Role: FOLLOWER
I20260924 04:35:53.693506 2015 consensus_queue.cc:260] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.693495 2014 raft_consensus.cc:348] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.693599 2014 raft_consensus.cc:374] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:53.693558 2013 raft_consensus.cc:348] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.693624 2014 raft_consensus.cc:729] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e71f16f0f3f4ac0951e9639b46b5d57, State: Initialized, Role: FOLLOWER
I20260924 04:35:53.693665 2013 raft_consensus.cc:374] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:53.693696 2013 raft_consensus.cc:729] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82fa0b767d81497389abab6f3609b6d4, State: Initialized, Role: FOLLOWER
I20260924 04:35:53.693783 2015 ts_tablet_manager.cc:1428] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:53.693759 2014 consensus_queue.cc:260] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.693753 2034 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.693801 2013 consensus_queue.cc:260] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.693876 2034 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:53.694016 2014 ts_tablet_manager.cc:1428] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:35:53.694037 2013 ts_tablet_manager.cc:1428] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:35:53.694202 2021 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.694329 2021 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:53.694379 2019 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.694492 2019 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:53.795403 2019 raft_consensus.cc:482] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:35:53.795588 2019 raft_consensus.cc:504] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.797158 2019 leader_election.cc:288] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47271a5d3c47472480fd400480111974 (127.0.98.2:39807), 2e71f16f0f3f4ac0951e9639b46b5d57 (127.0.98.3:40817)
I20260924 04:35:53.802440 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.797915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40817 (local address 127.0.98.1:53743)
0924 04:35:53.798288 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.798304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.798440 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.798809 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.798814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.798839 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:53.798996 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.799004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.799935 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.799946 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.801096 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.801103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.801856 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.801860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.802016 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.802034 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.802124 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.802238 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":24,"spinlock_wait_cycles":18944,"thread_start_us":83,"threads_started":1}
I20260924 04:35:53.802618 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.797506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50135 (local address 127.0.98.2:39807)
0924 04:35:53.797637 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.797643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.797865 (+ 222us) server_negotiation.cc:407] Connection header received
0924 04:35:53.798092 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.798095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.798150 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.798201 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:53.798681 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.799443 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.801073 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.801841 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.801961 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.802047 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.802445 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.802512 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:35:53.802649 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.797376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39807 (local address 127.0.98.1:50135)
0924 04:35:53.797777 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.797791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.797960 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.798358 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.798363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.798383 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:53.798549 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.798556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.799612 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.799614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.800940 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.800950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.802112 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.802117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.802296 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.802327 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.802408 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.802478 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":25,"spinlock_wait_cycles":768,"thread_start_us":116,"threads_started":1}
I20260924 04:35:53.803442 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.797696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53743 (local address 127.0.98.3:40817)
0924 04:35:53.797861 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.797865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.798387 (+ 522us) server_negotiation.cc:407] Connection header received
0924 04:35:53.798576 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.798582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.798658 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.798731 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:53.799104 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.799796 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.801232 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.801741 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.801840 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.801913 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.802139 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.802217 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:35:53.803418 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19e3e76f56244de92679ceed9ff645e" candidate_uuid: "82fa0b767d81497389abab6f3609b6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47271a5d3c47472480fd400480111974" is_pre_election: true
I20260924 04:35:53.803673 1804 raft_consensus.cc:2454] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82fa0b767d81497389abab6f3609b6d4 in term 0.
I20260924 04:35:53.803872 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19e3e76f56244de92679ceed9ff645e" candidate_uuid: "82fa0b767d81497389abab6f3609b6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" is_pre_election: true
I20260924 04:35:53.804039 1935 raft_consensus.cc:2454] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82fa0b767d81497389abab6f3609b6d4 in term 0.
I20260924 04:35:53.804291 1598 leader_election.cc:302] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47271a5d3c47472480fd400480111974, 82fa0b767d81497389abab6f3609b6d4; no voters:
I20260924 04:35:53.804548 2019 raft_consensus.cc:2781] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:35:53.804625 2019 raft_consensus.cc:482] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:35:53.804678 2019 raft_consensus.cc:3037] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:53.805805 2019 raft_consensus.cc:504] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.806241 2019 leader_election.cc:288] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [CANDIDATE]: Term 1 election: Requested vote from peers 47271a5d3c47472480fd400480111974 (127.0.98.2:39807), 2e71f16f0f3f4ac0951e9639b46b5d57 (127.0.98.3:40817)
I20260924 04:35:53.806653 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19e3e76f56244de92679ceed9ff645e" candidate_uuid: "82fa0b767d81497389abab6f3609b6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57"
I20260924 04:35:53.806767 1935 raft_consensus.cc:3037] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:53.807377 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19e3e76f56244de92679ceed9ff645e" candidate_uuid: "82fa0b767d81497389abab6f3609b6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47271a5d3c47472480fd400480111974"
I20260924 04:35:53.807554 1804 raft_consensus.cc:3037] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:53.808041 1935 raft_consensus.cc:2454] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82fa0b767d81497389abab6f3609b6d4 in term 1.
I20260924 04:35:53.808399 1595 leader_election.cc:302] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e71f16f0f3f4ac0951e9639b46b5d57, 82fa0b767d81497389abab6f3609b6d4; no voters:
I20260924 04:35:53.808584 2019 raft_consensus.cc:2781] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:53.808727 1804 raft_consensus.cc:2454] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82fa0b767d81497389abab6f3609b6d4 in term 1.
I20260924 04:35:53.808774 2019 raft_consensus.cc:686] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 82fa0b767d81497389abab6f3609b6d4, State: Running, Role: LEADER
I20260924 04:35:53.809078 2019 consensus_queue.cc:237] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } }
I20260924 04:35:53.810453 2055 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: TxnStatusTablet state changed. Reason: New leader 82fa0b767d81497389abab6f3609b6d4. Latest consensus state: current_term: 1 leader_uuid: "82fa0b767d81497389abab6f3609b6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.810621 2055 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:35:53.811213 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:35:53.811380 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:35:53.811758 1507 catalog_manager.cc:5495] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82fa0b767d81497389abab6f3609b6d4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "82fa0b767d81497389abab6f3609b6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } health_report { overall_health: UNKNOWN } } }
I20260924 04:35:53.848592 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.844539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46373 (local address 127.0.0.1:40754)
0924 04:35:53.844688 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.844704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.844798 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.845125 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.845127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.845171 (+ 44us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:35:53.845268 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.845273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.846277 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.846280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.846880 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.846888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.847432 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.847434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.847496 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.847536 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:35:53.848354 (+ 818us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:35:53.848366 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.848416 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.848483 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:35:53.848634 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.844657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.0.98.1:46373)
0924 04:35:53.844780 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.844785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.844799 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:35:53.844933 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.844937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.845006 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.845115 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:53.845373 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.846162 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.846984 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.847328 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.847430 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.847722 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:53.848255 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:53.848366 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.848429 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.848487 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:35:53.851439 1935 raft_consensus.cc:1261] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Refusing update from remote peer 82fa0b767d81497389abab6f3609b6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:53.851661 1804 raft_consensus.cc:1261] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974 [term 1 FOLLOWER]: Refusing update from remote peer 82fa0b767d81497389abab6f3609b6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:53.851908 2055 consensus_queue.cc:1035] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:53.852159 2019 consensus_queue.cc:1035] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:53.853560 2021 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: TxnStatusTablet state changed. Reason: New leader 82fa0b767d81497389abab6f3609b6d4. Latest consensus state: current_term: 1 leader_uuid: "82fa0b767d81497389abab6f3609b6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.853674 2021 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:53.854892 2034 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: TxnStatusTablet state changed. Reason: New leader 82fa0b767d81497389abab6f3609b6d4. Latest consensus state: current_term: 1 leader_uuid: "82fa0b767d81497389abab6f3609b6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.855011 2034 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:53.856066 2029 mvcc.cc:205] Tried to move back new op lower bound from 7332759772570599424 to 7332759772405059584. Current Snapshot: MvccSnapshot[applied={T|T < 7332759772570599424}]
I20260924 04:35:53.857303 2055 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82fa0b767d81497389abab6f3609b6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.857415 2055 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:35:53.857515 2019 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82fa0b767d81497389abab6f3609b6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.857613 2019 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 82fa0b767d81497389abab6f3609b6d4: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:35:53.858158 2021 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82fa0b767d81497389abab6f3609b6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.858283 2021 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 2e71f16f0f3f4ac0951e9639b46b5d57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:53.861109 2034 tablet_replica.cc:439] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82fa0b767d81497389abab6f3609b6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 } } peers { permanent_uuid: "47271a5d3c47472480fd400480111974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39807 } } peers { permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 } } }
I20260924 04:35:53.861258 2034 tablet_replica.cc:442] T c19e3e76f56244de92679ceed9ff645e P 47271a5d3c47472480fd400480111974: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:53.876746 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.869303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53019 (local address 127.0.98.1:46373)
0924 04:35:53.869429 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.869433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.873447 (+ 4014us) server_negotiation.cc:407] Connection header received
0924 04:35:53.873576 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.873578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.873639 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.873703 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:53.873951 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.874431 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.875674 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.875881 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.875964 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.876266 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:53.876387 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:53.876441 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.876536 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.876582 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:35:53.876791 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.869230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46373 (local address 127.0.98.2:53019)
0924 04:35:53.873352 (+ 4122us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.873368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.873479 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.873735 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.873738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.873747 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:35:53.873855 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.873861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.874528 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.874531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.875550 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.875562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.875981 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.875985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.876080 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.876137 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:35:53.876450 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:35:53.876456 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.876527 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.876628 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4016,"thread_start_us":96,"threads_started":1}
I20260924 04:35:53.883822 1935 raft_consensus.cc:1261] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Refusing update from remote peer 47271a5d3c47472480fd400480111974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:53.883865 1664 raft_consensus.cc:1261] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Refusing update from remote peer 47271a5d3c47472480fd400480111974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:53.884222 2034 consensus_queue.cc:1035] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e71f16f0f3f4ac0951e9639b46b5d57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:53.884393 2024 consensus_queue.cc:1035] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82fa0b767d81497389abab6f3609b6d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:53.887454 2027 mvcc.cc:205] Tried to move back new op lower bound from 7332759772704800768 to 7332759771664216064. Current Snapshot: MvccSnapshot[applied={T|T < 7332759772704800768}]
I20260924 04:35:53.907884 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.904044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47817 (local address 127.0.98.2:39807)
0924 04:35:53.904163 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.904168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.904443 (+ 275us) server_negotiation.cc:407] Connection header received
0924 04:35:53.904584 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.904586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.904647 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.904725 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:53.905192 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.905898 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.906845 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.907078 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.907191 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.907455 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:53.907532 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:53.907605 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.907695 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.907748 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:35:53.908465 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.904002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39807 (local address 127.0.98.1:47817)
0924 04:35:53.904362 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:53.904374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:53.904488 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:53.904873 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:53.904876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:53.904884 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:35:53.905031 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.905038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.906027 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.906030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.906726 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:53.906735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.907187 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.907191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:53.907293 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.907330 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:35:53.907618 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:35:53.907627 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 04:35:53.907687 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:35:53.908316 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
I20260924 04:35:53.930696 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:53.930860 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:53.932662 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e23998c5a9c447cac2b2fa9a9293e29: Sending DeleteTablet for 3 replicas of tablet 52c0345f598a48debc6f26dad8b05655
I20260924 04:35:53.933203 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c0345f598a48debc6f26dad8b05655 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40752
I20260924 04:35:53.933262 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c0345f598a48debc6f26dad8b05655 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50758
I20260924 04:35:53.933413 2039 tablet_replica.cc:330] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: stopping tablet replica
I20260924 04:35:53.933426 2038 tablet_replica.cc:330] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: stopping tablet replica
I20260924 04:35:53.933529 2039 raft_consensus.cc:2229] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:35:53.933531 2038 raft_consensus.cc:2229] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:35:53.933670 2039 raft_consensus.cc:2258] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:53.933722 2038 raft_consensus.cc:2258] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:53.933841 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 52c0345f598a48debc6f26dad8b05655 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52804
I20260924 04:35:53.934007 2037 tablet_replica.cc:330] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: stopping tablet replica
I20260924 04:35:53.934135 2037 raft_consensus.cc:2229] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:35:53.934244 2037 raft_consensus.cc:2258] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:53.934401 2038 ts_tablet_manager.cc:1905] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:53.934587 2039 ts_tablet_manager.cc:1905] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:53.936753 2039 ts_tablet_manager.cc:1918] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:35:53.936846 2039 log.cc:1196] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/52c0345f598a48debc6f26dad8b05655
I20260924 04:35:53.937297 2039 ts_tablet_manager.cc:1939] T 52c0345f598a48debc6f26dad8b05655 P 82fa0b767d81497389abab6f3609b6d4: Deleting consensus metadata
I20260924 04:35:53.937672 2038 ts_tablet_manager.cc:1918] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:35:53.937741 2038 log.cc:1196] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/52c0345f598a48debc6f26dad8b05655
I20260924 04:35:53.937994 2038 ts_tablet_manager.cc:1939] T 52c0345f598a48debc6f26dad8b05655 P 47271a5d3c47472480fd400480111974: Deleting consensus metadata
I20260924 04:35:53.937978 1492 catalog_manager.cc:4841] TS 82fa0b767d81497389abab6f3609b6d4 (127.0.98.1:46373): tablet 52c0345f598a48debc6f26dad8b05655 (table test-table [id=e64cdc9a2d8e4b8db65bb36670981161]) successfully deleted
I20260924 04:35:53.938602 1494 catalog_manager.cc:4841] TS 47271a5d3c47472480fd400480111974 (127.0.98.2:39807): tablet 52c0345f598a48debc6f26dad8b05655 (table test-table [id=e64cdc9a2d8e4b8db65bb36670981161]) successfully deleted
I20260924 04:35:53.939127 2037 ts_tablet_manager.cc:1905] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:53.941897 2037 ts_tablet_manager.cc:1918] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:35:53.941978 2037 log.cc:1196] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/52c0345f598a48debc6f26dad8b05655
I20260924 04:35:53.942251 2037 ts_tablet_manager.cc:1939] T 52c0345f598a48debc6f26dad8b05655 P 2e71f16f0f3f4ac0951e9639b46b5d57: Deleting consensus metadata
I20260924 04:35:53.942910 1491 catalog_manager.cc:4841] TS 2e71f16f0f3f4ac0951e9639b46b5d57 (127.0.98.3:40817): tablet 52c0345f598a48debc6f26dad8b05655 (table test-table [id=e64cdc9a2d8e4b8db65bb36670981161]) successfully deleted
Sep 24 04:35:53 dist-test-slave-jlnt krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224553, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:35:53.949782 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.938645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.0.98.1:46373)
0924 04:35:53.938754 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.938758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.939274 (+ 516us) server_negotiation.cc:407] Connection header received
0924 04:35:53.939474 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.939477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.939529 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.939634 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:53.940360 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.940954 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.941847 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.942164 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.942274 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.945751 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:53.945760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:53.945774 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:53.945801 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:53.947628 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.948019 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.948026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.948028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.948131 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.948371 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.948374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.948376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.949053 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:53.949334 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.949435 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.949583 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:35:53.950254 1702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40770
I20260924 04:35:53.959121 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:53.951923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.0.98.62:40543)
0924 04:35:53.952052 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:53.952056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:53.952233 (+ 177us) server_negotiation.cc:407] Connection header received
0924 04:35:53.952385 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:53.952393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:53.952433 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:53.952513 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:53.953110 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.953672 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.954423 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:53.954646 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:53.954754 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:53.956026 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:53.956029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:53.956032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:53.956056 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:53.957567 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.957962 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.957967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.957969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.958045 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:53.958309 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:53.958313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:53.958315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:53.958480 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:53.958611 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:53.958847 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 04:35:53.958961 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 04:35:53.959470 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52304
I20260924 04:35:53.960700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 1579
I20260924 04:35:53.968988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 1719
I20260924 04:35:53.975507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 1859
I20260924 04:35:53.981031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 1475
2026-09-24T04:35:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2913 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:35:54 dist-test-slave-jlnt krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:35:54 dist-test-slave-jlnt krb5kdc[2077](info): set up 2 sockets
Sep 24 04:35:54 dist-test-slave-jlnt krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260924 04:35:55.032874 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:35:55 dist-test-slave-jlnt krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:35:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:35:55.152614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35623
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:55.262274 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:55.262557 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:55.262629 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:55.266678 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:35:55.266757 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:35:55.266788 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:55.266830 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:55.266865 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:35:55.266896 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:35:55.271966 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35623
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:55.272969 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:55.273963 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:55.282066 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:55.282027 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:55.282162 2090 server_base.cc:1034] running on GCE node
W20260924 04:35:55.282091 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:55.282795 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:55.283773 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:55.284931 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790224555284922 us; error 33 us; skew 500 ppm
Sep 24 04:35:55 dist-test-slave-jlnt krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:55.288102 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:35:55.288964 2090 webserver.cc:460] Webserver started at http://127.0.98.62:32979/ using document root <none> and password file <none>
I20260924 04:35:55.289538 2090 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:55.289599 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:55.289760 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:55.291649 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "3e2d389a39934923bdf8d0b3fce0afd1"
format_stamp: "Formatted at 2026-09-24 04:35:55 on dist-test-slave-jlnt"
server_key: "5e1bc62ad3a7af48a951e880b27f3ce7"
server_key_iv: "a2e299a74263f3b9bb9987532f268478"
server_key_version: "encryptionkey@0"
I20260924 04:35:55.292143 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e2d389a39934923bdf8d0b3fce0afd1"
format_stamp: "Formatted at 2026-09-24 04:35:55 on dist-test-slave-jlnt"
server_key: "5e1bc62ad3a7af48a951e880b27f3ce7"
server_key_iv: "a2e299a74263f3b9bb9987532f268478"
server_key_version: "encryptionkey@0"
I20260924 04:35:55.295753 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:35:55.298041 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:55.299228 2090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:35:55.299368 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "3e2d389a39934923bdf8d0b3fce0afd1"
format_stamp: "Formatted at 2026-09-24 04:35:55 on dist-test-slave-jlnt"
server_key: "5e1bc62ad3a7af48a951e880b27f3ce7"
server_key_iv: "a2e299a74263f3b9bb9987532f268478"
server_key_version: "encryptionkey@0"
I20260924 04:35:55.299476 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:55.324589 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:55.328630 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:55.328825 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:55.336293 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35623
I20260924 04:35:55.336318 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35623 every 8 connection(s)
I20260924 04:35:55.337200 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:35:55.338186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 2090
I20260924 04:35:55.338301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:35:55.338588 392 external_mini_cluster.cc:1428] Setting key 7431ec00f98d8562837bc2aa985516cd
I20260924 04:35:55.340409 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:55.345876 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: Bootstrap starting.
Sep 24 04:35:55 dist-test-slave-jlnt krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:55.348029 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:55.348886 2168 log.cc:824] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:55.351135 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: No bootstrap required, opened a new log
I20260924 04:35:55.354072 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2d389a39934923bdf8d0b3fce0afd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35623 } }
I20260924 04:35:55.354305 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:55.354393 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2d389a39934923bdf8d0b3fce0afd1, State: Initialized, Role: FOLLOWER
I20260924 04:35:55.354466 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:55.339953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.98.62:35623)
0924 04:35:55.340378 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:55.340393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:55.340426 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:35:55.341082 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:55.341098 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:55.341477 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:55.341813 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:55.342593 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.343545 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.344391 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.344862 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.345037 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:55.348101 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:55.348106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:55.348139 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:55.348170 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:55.351728 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.352176 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.352181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.352185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.352287 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.352576 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.352579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.352581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.352949 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:55.353143 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:55.353442 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:35:55.353715 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260924 04:35:55.354869 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2d389a39934923bdf8d0b3fce0afd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35623 } }
I20260924 04:35:55.354990 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:35:55.355067 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:35:55.355161 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:55.356204 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2d389a39934923bdf8d0b3fce0afd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35623 } }
I20260924 04:35:55.356552 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e2d389a39934923bdf8d0b3fce0afd1; no voters:
I20260924 04:35:55.356891 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:35:55.357014 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:55.357232 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [term 1 LEADER]: Becoming Leader. State: Replica: 3e2d389a39934923bdf8d0b3fce0afd1, State: Running, Role: LEADER
I20260924 04:35:55.357546 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2d389a39934923bdf8d0b3fce0afd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35623 } }
I20260924 04:35:55.358085 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:35:55.359653 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e2d389a39934923bdf8d0b3fce0afd1. Latest consensus state: current_term: 1 leader_uuid: "3e2d389a39934923bdf8d0b3fce0afd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2d389a39934923bdf8d0b3fce0afd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35623 } } }
I20260924 04:35:55.359790 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:35:55.359968 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e2d389a39934923bdf8d0b3fce0afd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2d389a39934923bdf8d0b3fce0afd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35623 } } }
I20260924 04:35:55.360256 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1 [sys.catalog]: This master's current role is: LEADER
W20260924 04:35:55.362805 2189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:35:55.362900 2189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:35:55.362982 2188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:35:55.363739 2188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:35:55.368556 2188 catalog_manager.cc:1347] Generated new cluster ID: 1ae9633db5ae4dc5adbc1b81938202c1
I20260924 04:35:55.368619 2188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:35:55.386459 2188 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:35:55.387770 2188 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:35:55.395874 2188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: Generated new TSK 0
I20260924 04:35:55.396674 2188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:35:55.433225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35623
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:55.554544 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:55.554796 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:55.554843 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:55.559167 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:55.559217 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:55.559302 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:35:55.559393 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:35:55.559448 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:35:55.564718 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:55.565828 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:55.566852 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:55.576232 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:55.576323 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:55.576272 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:55.576565 2194 server_base.cc:1034] running on GCE node
I20260924 04:35:55.577018 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:55.577638 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:55.578830 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790224555578802 us; error 41 us; skew 500 ppm
Sep 24 04:35:55 dist-test-slave-jlnt krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:55.582324 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:35:55.583199 2194 webserver.cc:460] Webserver started at http://127.0.98.1:36915/ using document root <none> and password file <none>
I20260924 04:35:55.583839 2194 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:55.583928 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:55.584168 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:55.586194 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "7358f32ee76344fa896aa0db09c0aac6"
format_stamp: "Formatted at 2026-09-24 04:35:55 on dist-test-slave-jlnt"
server_key: "5bfe1aea7b9e38f6ae051d003c814e70"
server_key_iv: "5be78ed2e059cd1ff6b7798f3ea50a26"
server_key_version: "encryptionkey@0"
I20260924 04:35:55.586752 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7358f32ee76344fa896aa0db09c0aac6"
format_stamp: "Formatted at 2026-09-24 04:35:55 on dist-test-slave-jlnt"
server_key: "5bfe1aea7b9e38f6ae051d003c814e70"
server_key_iv: "5be78ed2e059cd1ff6b7798f3ea50a26"
server_key_version: "encryptionkey@0"
I20260924 04:35:55.590382 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:35:55.592751 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:55.594204 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:35:55.594358 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "7358f32ee76344fa896aa0db09c0aac6"
format_stamp: "Formatted at 2026-09-24 04:35:55 on dist-test-slave-jlnt"
server_key: "5bfe1aea7b9e38f6ae051d003c814e70"
server_key_iv: "5be78ed2e059cd1ff6b7798f3ea50a26"
server_key_version: "encryptionkey@0"
I20260924 04:35:55.594485 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:55.611081 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:55.615299 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:55.615509 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:55.616917 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:55.616990 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:55.617063 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:55.617105 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:55.627864 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34721
Sep 24 04:35:55 dist-test-slave-jlnt krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:55.628832 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:35:55.629696 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34721 every 8 connection(s)
I20260924 04:35:55.629799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 2194
I20260924 04:35:55.629922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:35:55.630244 392 external_mini_cluster.cc:1428] Setting key 71d430c051b412dc842f372a16ab645a
I20260924 04:35:55.633639 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:55.617760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57247 (local address 127.0.98.62:35623)
0924 04:35:55.617892 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:55.617895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:55.618953 (+ 1058us) server_negotiation.cc:407] Connection header received
0924 04:35:55.620384 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:55.620387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:55.620447 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:55.620562 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:55.622176 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.622784 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.624608 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.624951 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.625038 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:55.628693 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:55.628697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:55.628700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:55.628730 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:55.631577 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.632077 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.632083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.632085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.632174 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.632524 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.632529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.632531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.632712 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:55.632819 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:55.633339 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 04:35:55.633476 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:35:55.634429 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:55.618262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35623 (local address 127.0.98.1:57247)
0924 04:35:55.618773 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:55.618829 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:55.620139 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:55.620767 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:55.620778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:55.621278 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:55.621990 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:55.622012 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.622897 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.622904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:55.624469 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:55.624483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.625104 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.625110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:55.625366 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:55.626824 (+ 1458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:55.626839 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:55.628523 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:55.631698 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:55.631704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:55.631717 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:55.631975 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:55.632273 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:55.632276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:55.632278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:55.632422 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:55.632818 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:55.632825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:55.633060 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 04:35:55.633282 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 04:35:55.633580 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260924 04:35:55.635746 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35623
I20260924 04:35:55.635922 2325 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:55.636533 2325 heartbeater.cc:511] Master 127.0.98.62:35623 requested a full tablet report, sending...
I20260924 04:35:55.638144 2121 ts_manager.cc:194] Registered new tserver with Master: 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721)
I20260924 04:35:55.639778 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57247
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:35:55.646879 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:55.638073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46283 (local address 127.0.98.62:35623)
0924 04:35:55.638195 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:55.638201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:55.638214 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:35:55.638244 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:55.638248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:55.638289 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:55.638359 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:55.638770 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.639272 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.640106 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.640322 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.640431 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:55.642357 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:55.642360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:55.642362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:55.642382 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:55.645244 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.645658 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.645661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.645663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.645740 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.646088 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.646094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.646097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.646298 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:55.646382 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:55.646681 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:35:55.646772 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:35:55.647010 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:55.637798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35623 (local address 127.0.98.1:46283)
0924 04:35:55.638038 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:55.638052 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:55.638135 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:55.638373 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:55.638376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:55.638572 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:55.638658 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:55.638663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.639390 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.639394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:55.639985 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:55.640002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.640459 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.640463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:55.640582 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:55.641843 (+ 1261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:55.641860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:55.642226 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:55.645352 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:55.645355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:55.645357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:55.645552 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:55.645842 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:55.645845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:55.645847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:55.645964 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:55.646399 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:55.646403 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:55.646513 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 04:35:55.646631 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:35:55.646791 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:35:55.660305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35623
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:55.767978 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:55.768296 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:55.768380 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:55.773216 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:55.773289 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:55.773376 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:35:55.773414 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:35:55.773491 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:35:55.779140 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:55.780138 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:55.781080 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:35:55.790462 2334 server_base.cc:1034] running on GCE node
W20260924 04:35:55.790417 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:55.790428 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:55.790417 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:55.791131 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:55.791666 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:55.792819 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790224555792790 us; error 41 us; skew 500 ppm
Sep 24 04:35:55 dist-test-slave-jlnt krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:55.796437 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:35:55.797315 2334 webserver.cc:460] Webserver started at http://127.0.98.2:37159/ using document root <none> and password file <none>
I20260924 04:35:55.797947 2334 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:55.798007 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:55.798328 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:55.800278 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "f06143ea5f11404db273399e823ec71e"
format_stamp: "Formatted at 2026-09-24 04:35:55 on dist-test-slave-jlnt"
server_key: "b0c21730b53efe6b2902a2dca24f187d"
server_key_iv: "e680a21a96257a8b9f4984f83b7f1d73"
server_key_version: "encryptionkey@0"
I20260924 04:35:55.800815 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f06143ea5f11404db273399e823ec71e"
format_stamp: "Formatted at 2026-09-24 04:35:55 on dist-test-slave-jlnt"
server_key: "b0c21730b53efe6b2902a2dca24f187d"
server_key_iv: "e680a21a96257a8b9f4984f83b7f1d73"
server_key_version: "encryptionkey@0"
I20260924 04:35:55.804585 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:35:55.806958 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:55.808130 2334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:35:55.808257 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "f06143ea5f11404db273399e823ec71e"
format_stamp: "Formatted at 2026-09-24 04:35:55 on dist-test-slave-jlnt"
server_key: "b0c21730b53efe6b2902a2dca24f187d"
server_key_iv: "e680a21a96257a8b9f4984f83b7f1d73"
server_key_version: "encryptionkey@0"
I20260924 04:35:55.808359 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:55.831565 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:55.837116 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:55.837391 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:55.838838 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:55.838914 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:55.838976 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:55.839030 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:55.849036 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42955
I20260924 04:35:55.849066 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42955 every 8 connection(s)
Sep 24 04:35:55 dist-test-slave-jlnt krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:55.850003 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:35:55.855086 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:55.839691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50811 (local address 127.0.98.62:35623)
0924 04:35:55.839840 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:55.839845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:55.840729 (+ 884us) server_negotiation.cc:407] Connection header received
0924 04:35:55.841883 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:55.841889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:55.841955 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:55.842065 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:55.843596 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.844508 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.846034 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.846305 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.846419 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:55.849994 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:55.849998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:55.850001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:55.850027 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:55.853021 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.853569 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.853571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.853573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.853655 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.854081 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.854083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.854085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.854255 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:55.854330 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:55.854846 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 04:35:55.854953 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:35:55.855790 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:55.840121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35623 (local address 127.0.98.2:50811)
0924 04:35:55.840544 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:55.840593 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:55.841634 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:55.842285 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:55.842295 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:55.842717 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:55.843407 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:55.843427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.844669 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.844672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:55.845899 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:55.845909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.846454 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.846458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:55.846644 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:55.848155 (+ 1511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:55.848170 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:55.849833 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:55.853175 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:55.853185 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:55.853201 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:55.853466 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:55.853747 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:55.853750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:55.853752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:55.853976 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:55.854365 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:55.854370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:55.854590 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 04:35:55.854742 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 04:35:55.854984 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":120,"threads_started":1}
I20260924 04:35:55.856393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 2334
I20260924 04:35:55.856495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:35:55.856786 392 external_mini_cluster.cc:1428] Setting key 9ae83d1a9f14d441032888f688653257
I20260924 04:35:55.857116 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35623
I20260924 04:35:55.857311 2465 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:55.857856 2465 heartbeater.cc:511] Master 127.0.98.62:35623 requested a full tablet report, sending...
I20260924 04:35:55.858950 2122 ts_manager.cc:194] Registered new tserver with Master: f06143ea5f11404db273399e823ec71e (127.0.98.2:42955)
I20260924 04:35:55.859768 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50811
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:35:55.868746 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:55.859493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35623 (local address 127.0.98.2:60181)
0924 04:35:55.859775 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:55.859792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:55.859936 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:55.860301 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:55.860488 (+ 187us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:55.860717 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:55.860873 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:55.860879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.861681 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.861684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:55.862231 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:55.862238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.862684 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.862688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:55.862783 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:55.863903 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:55.863915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:55.864260 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:55.867369 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:55.867372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:55.867373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:55.867559 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:55.867778 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:55.867780 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:55.867781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:55.867841 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:55.868290 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:55.868293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:55.868376 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 04:35:55.868483 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:35:55.868604 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260924 04:35:55.868857 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:55.859300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60181 (local address 127.0.98.62:35623)
0924 04:35:55.859398 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:55.859402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:55.859860 (+ 458us) server_negotiation.cc:407] Connection header received
0924 04:35:55.860048 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:55.860053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:55.860104 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:55.860154 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:55.860984 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.861559 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.862342 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:55.862570 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:55.862672 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:55.864420 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:55.864424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:55.864427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:55.864450 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:55.867270 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.867646 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.867648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.867649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.867719 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:55.867890 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:55.867892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:55.867893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:55.868178 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:55.868260 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:55.868581 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 04:35:55.868708 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:35:55.885247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35623
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:35:55.995122 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:35:55.995368 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:35:55.995416 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:35:55.999591 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:35:55.999647 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:35:55.999686 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:35:55.999737 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:35:55.999804 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:35:56.005096 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:35:56.006166 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:35:56.007174 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:35:56.016888 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:56.016872 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:35:56.016979 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:35:56.017164 2474 server_base.cc:1034] running on GCE node
I20260924 04:35:56.017598 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:35:56.018239 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:35:56.019429 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790224556019418 us; error 27 us; skew 500 ppm
Sep 24 04:35:56 dist-test-slave-jlnt krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:35:56.022891 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:35:56.023854 2474 webserver.cc:460] Webserver started at http://127.0.98.3:37899/ using document root <none> and password file <none>
I20260924 04:35:56.024612 2474 fs_manager.cc:359] Metadata directory not provided
I20260924 04:35:56.024708 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:35:56.024945 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:35:56.027060 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "771c5c02ea42408d9241d2fc916dbcd0"
format_stamp: "Formatted at 2026-09-24 04:35:56 on dist-test-slave-jlnt"
server_key: "5a8c0fe57bce441bb251219991e40ef7"
server_key_iv: "b9219e71fe69f55b46132a8a2a258f72"
server_key_version: "encryptionkey@0"
I20260924 04:35:56.027637 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "771c5c02ea42408d9241d2fc916dbcd0"
format_stamp: "Formatted at 2026-09-24 04:35:56 on dist-test-slave-jlnt"
server_key: "5a8c0fe57bce441bb251219991e40ef7"
server_key_iv: "b9219e71fe69f55b46132a8a2a258f72"
server_key_version: "encryptionkey@0"
I20260924 04:35:56.031492 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:35:56.034058 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:56.035338 2474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:35:56.035461 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "771c5c02ea42408d9241d2fc916dbcd0"
format_stamp: "Formatted at 2026-09-24 04:35:56 on dist-test-slave-jlnt"
server_key: "5a8c0fe57bce441bb251219991e40ef7"
server_key_iv: "b9219e71fe69f55b46132a8a2a258f72"
server_key_version: "encryptionkey@0"
I20260924 04:35:56.035574 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:35:56.047983 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:35:56.052155 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:35:56.052358 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:35:56.053917 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:35:56.053988 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:35:56.054054 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:35:56.054108 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:35:56 dist-test-slave-jlnt krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:56.065068 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36077
I20260924 04:35:56.066325 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:35:56.067221 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36077 every 8 connection(s)
I20260924 04:35:56.069523 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.054633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45169 (local address 127.0.98.62:35623)
0924 04:35:56.054793 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.054797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.055596 (+ 799us) server_negotiation.cc:407] Connection header received
0924 04:35:56.056530 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.056533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.056584 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.056687 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:56.058208 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.058982 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.060512 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.060770 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.060906 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.064407 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:56.064411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:56.064414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:56.064441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:56.067526 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:56.068022 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:56.068026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:56.068029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:56.068106 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:56.068458 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:56.068461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:56.068462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:56.068644 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:56.068750 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.069221 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.069336 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:35:56.070325 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.055012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35623 (local address 127.0.98.3:45169)
0924 04:35:56.055429 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.055476 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.056328 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.056859 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.056867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.057392 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:56.058071 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.058086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.059102 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.059105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.060388 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.060397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.060951 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.060955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.061206 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.062527 (+ 1321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:56.062550 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:56.064251 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:56.067638 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:56.067644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:56.067658 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:56.067910 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:56.068207 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:56.068210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:56.068212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:56.068356 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:56.068745 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:56.068751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:56.068978 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.069182 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.069495 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":117,"threads_started":1}
I20260924 04:35:56.071357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 2474
I20260924 04:35:56.071489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:35:56.071707 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35623
I20260924 04:35:56.071831 392 external_mini_cluster.cc:1428] Setting key 70a625cf51e46e31987b0bb3bbce24dd
I20260924 04:35:56.071905 2605 heartbeater.cc:464] Registering TS with master...
I20260924 04:35:56.072703 2605 heartbeater.cc:511] Master 127.0.98.62:35623 requested a full tablet report, sending...
I20260924 04:35:56.073717 2122 ts_manager.cc:194] Registered new tserver with Master: 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3:36077)
I20260924 04:35:56.074482 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45169
I20260924 04:35:56.082382 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.073989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35623 (local address 127.0.98.3:60333)
0924 04:35:56.074256 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.074270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.074360 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.074696 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.074698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.074929 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:56.075021 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.075034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.075982 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.075984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.076574 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.076582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.076989 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.076991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.077056 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.078366 (+ 1310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:56.078377 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:56.078687 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:56.081026 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:56.081029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:56.081031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:56.081305 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:56.081584 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:56.081586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:56.081587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:56.081678 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:56.081986 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:56.081988 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:56.082059 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.082139 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.082243 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":17,"thread_start_us":105,"threads_started":1}
I20260924 04:35:56.082384 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.074045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60333 (local address 127.0.98.62:35623)
0924 04:35:56.074170 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.074174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.074324 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:35:56.074434 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.074436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.074470 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.074686 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:56.075136 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.075859 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.076687 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.076891 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.076973 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.078806 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:56.078809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:56.078811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:56.078831 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:56.080929 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:56.081394 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:56.081398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:56.081399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:56.081486 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:56.081753 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:56.081755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:56.081756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:56.081909 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:56.081986 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.082193 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.082281 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:35:56.085665 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:35:56 dist-test-slave-jlnt krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:35:56 dist-test-slave-jlnt krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:56.112427 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.102515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.0.98.62:35623)
0924 04:35:56.102681 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.102684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.102696 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:35:56.102734 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.102737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.102772 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.102846 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:56.103476 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.104039 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.104835 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.105038 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.105113 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.107911 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:56.107917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:56.107924 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:56.107945 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:56.110724 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:56.111227 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:56.111233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:56.111234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:56.111315 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:56.111586 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:56.111589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:56.111591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:56.111760 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:56.111892 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.112150 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.112261 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:35:56.115978 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:35:56.119077 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:35:56.133002 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.127979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.0.98.1:34721)
0924 04:35:56.128354 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.128372 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.128398 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:35:56.128473 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.128478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.128699 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.128858 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:56.129130 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.130045 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.131649 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.132344 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.132465 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.132621 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.132752 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.132839 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":89,"threads_started":1}
I20260924 04:35:56.132997 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.127247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34721 (local address 127.0.0.1:50402)
0924 04:35:56.127843 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.127858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.127945 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.128850 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.128856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.128882 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:56.129008 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.129014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.130157 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.130162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.131521 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.131531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.132468 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.132471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.132572 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.132607 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.132722 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.132838 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":21,"thread_start_us":120,"threads_started":1}
I20260924 04:35:56.133275 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.127203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42955 (local address 127.0.0.1:53610)
0924 04:35:56.127631 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.127677 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.127800 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.128979 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.128984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.129003 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:56.129141 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.129171 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.130678 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.130681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.131774 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.131780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.132834 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.132838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.132924 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.132944 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.132996 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.133076 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":21,"thread_start_us":84,"threads_started":1}
I20260924 04:35:56.134004 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.127903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36077 (local address 127.0.0.1:60642)
0924 04:35:56.128165 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.128180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.128269 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.129290 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.129293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.129309 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:56.129386 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.129391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.130722 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.130725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.131903 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.131913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.133015 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.133018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.133701 (+ 683us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.133720 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.133814 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.133881 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":3,"thread_start_us":133,"threads_started":1}
I20260924 04:35:56.134212 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.128394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.0.98.3:36077)
0924 04:35:56.128814 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.128825 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.128846 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:35:56.128912 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.128917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.129119 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.129270 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:56.129509 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.130603 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.132021 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.132738 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.132833 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.132961 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.133848 (+ 887us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.133945 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":118,"threads_started":1}
I20260924 04:35:56.133860 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.127301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.0.98.2:42955)
0924 04:35:56.128526 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.128537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.128556 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:35:56.128617 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.128621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.128835 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.129248 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:56.129382 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.130554 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.131884 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.132691 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.133458 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.133586 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.133632 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.133710 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1138,"thread_start_us":99,"threads_started":1}
I20260924 04:35:56.136168 2259 tablet_service.cc:1467] Processing CreateTablet for tablet e25f0e9fee994f88a04f5d407e0ac25e (DEFAULT_TABLE table=test-table [id=529d22bf9304494f8f6d07df342235ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:56.137286 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25f0e9fee994f88a04f5d407e0ac25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:56.136862 2521 tablet_service.cc:1467] Processing CreateTablet for tablet e25f0e9fee994f88a04f5d407e0ac25e (DEFAULT_TABLE table=test-table [id=529d22bf9304494f8f6d07df342235ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:56.137414 2394 tablet_service.cc:1467] Processing CreateTablet for tablet e25f0e9fee994f88a04f5d407e0ac25e (DEFAULT_TABLE table=test-table [id=529d22bf9304494f8f6d07df342235ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:56.137903 2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25f0e9fee994f88a04f5d407e0ac25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:56.138267 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25f0e9fee994f88a04f5d407e0ac25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:56.144172 2628 tablet_bootstrap.cc:492] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: Bootstrap starting.
I20260924 04:35:56.145434 2629 tablet_bootstrap.cc:492] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: Bootstrap starting.
I20260924 04:35:56.146397 2628 tablet_bootstrap.cc:654] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:56.146379 2630 tablet_bootstrap.cc:492] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: Bootstrap starting.
I20260924 04:35:56.147109 2628 log.cc:824] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:56.147920 2629 tablet_bootstrap.cc:654] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:56.148319 2630 tablet_bootstrap.cc:654] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:56.148718 2629 log.cc:824] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:56.149068 2628 tablet_bootstrap.cc:492] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: No bootstrap required, opened a new log
I20260924 04:35:56.149102 2630 log.cc:824] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:35:56.149298 2628 ts_tablet_manager.cc:1397] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:35:56.150626 2629 tablet_bootstrap.cc:492] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: No bootstrap required, opened a new log
I20260924 04:35:56.150822 2629 ts_tablet_manager.cc:1397] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:35:56.150988 2630 tablet_bootstrap.cc:492] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: No bootstrap required, opened a new log
I20260924 04:35:56.151168 2630 ts_tablet_manager.cc:1397] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 04:35:56.153224 2628 raft_consensus.cc:348] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.153491 2628 raft_consensus.cc:374] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:56.153554 2628 raft_consensus.cc:729] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06143ea5f11404db273399e823ec71e, State: Initialized, Role: FOLLOWER
I20260924 04:35:56.153594 2629 raft_consensus.cc:348] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.153847 2629 raft_consensus.cc:374] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:56.153913 2629 raft_consensus.cc:729] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7358f32ee76344fa896aa0db09c0aac6, State: Initialized, Role: FOLLOWER
I20260924 04:35:56.153982 2628 consensus_queue.cc:260] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.154313 2629 consensus_queue.cc:260] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.154709 2465 heartbeater.cc:503] Master 127.0.98.62:35623 was elected leader, sending a full tablet report...
I20260924 04:35:56.154961 2628 ts_tablet_manager.cc:1428] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:35:56.155158 2629 ts_tablet_manager.cc:1428] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 04:35:56.155464 2325 heartbeater.cc:503] Master 127.0.98.62:35623 was elected leader, sending a full tablet report...
I20260924 04:35:56.155452 2630 raft_consensus.cc:348] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.155674 2630 raft_consensus.cc:374] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:56.155740 2630 raft_consensus.cc:729] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771c5c02ea42408d9241d2fc916dbcd0, State: Initialized, Role: FOLLOWER
I20260924 04:35:56.156092 2630 consensus_queue.cc:260] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.156996 2630 ts_tablet_manager.cc:1428] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 04:35:56.157346 2605 heartbeater.cc:503] Master 127.0.98.62:35623 was elected leader, sending a full tablet report...
I20260924 04:35:56.316648 2636 raft_consensus.cc:482] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:35:56.316865 2636 raft_consensus.cc:504] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
W20260924 04:35:56.317318 2606 tablet.cc:2369] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:35:56.318259 2636 leader_election.cc:288] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f06143ea5f11404db273399e823ec71e (127.0.98.2:42955), 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721)
I20260924 04:35:56.322435 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.318578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45053 (local address 127.0.98.2:42955)
0924 04:35:56.318688 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.318693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.318825 (+ 132us) server_negotiation.cc:407] Connection header received
0924 04:35:56.318903 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.318906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.318980 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.319079 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:56.319377 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.319995 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.321352 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.321934 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.322014 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.322064 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.322279 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.322338 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:35:56.322463 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.318476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42955 (local address 127.0.98.3:45053)
0924 04:35:56.318676 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.318692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.318795 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.319099 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.319101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.319113 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:56.319267 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.319275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.320113 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.320115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.321210 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.321219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.322048 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.322051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.322157 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.322182 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.322260 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.322331 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 04:35:56.323005 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.318693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40819 (local address 127.0.98.1:34721)
0924 04:35:56.318839 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.318845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.318973 (+ 128us) server_negotiation.cc:407] Connection header received
0924 04:35:56.319046 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.319051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.319099 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.319170 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:56.319812 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.320434 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.321834 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.322399 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.322505 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.322565 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.322824 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.322898 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:35:56.323158 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f0e9fee994f88a04f5d407e0ac25e" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06143ea5f11404db273399e823ec71e" is_pre_election: true
I20260924 04:35:56.323362 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.318552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34721 (local address 127.0.98.3:40819)
0924 04:35:56.318879 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.318891 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.318973 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.319188 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.319190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.319201 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:56.319695 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.319700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.320571 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.320574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.321694 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.321704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.322538 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.322543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.322649 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.322680 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.323163 (+ 483us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.323237 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
I20260924 04:35:56.323467 2419 raft_consensus.cc:2454] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 0.
I20260924 04:35:56.323974 2490 leader_election.cc:302] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771c5c02ea42408d9241d2fc916dbcd0, f06143ea5f11404db273399e823ec71e; no voters:
I20260924 04:35:56.324025 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f0e9fee994f88a04f5d407e0ac25e" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358f32ee76344fa896aa0db09c0aac6" is_pre_election: true
I20260924 04:35:56.324267 2279 raft_consensus.cc:2454] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 0.
I20260924 04:35:56.324272 2636 raft_consensus.cc:2781] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:35:56.324368 2636 raft_consensus.cc:482] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:35:56.324429 2636 raft_consensus.cc:3037] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:56.325711 2636 raft_consensus.cc:504] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.326082 2636 leader_election.cc:288] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 election: Requested vote from peers f06143ea5f11404db273399e823ec71e (127.0.98.2:42955), 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721)
I20260924 04:35:56.326409 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f0e9fee994f88a04f5d407e0ac25e" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06143ea5f11404db273399e823ec71e"
I20260924 04:35:56.326445 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f0e9fee994f88a04f5d407e0ac25e" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358f32ee76344fa896aa0db09c0aac6"
I20260924 04:35:56.326519 2419 raft_consensus.cc:3037] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:56.326617 2279 raft_consensus.cc:3037] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:56.327559 2419 raft_consensus.cc:2454] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 1.
I20260924 04:35:56.327639 2279 raft_consensus.cc:2454] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 1.
I20260924 04:35:56.327955 2492 leader_election.cc:302] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7358f32ee76344fa896aa0db09c0aac6, 771c5c02ea42408d9241d2fc916dbcd0; no voters:
I20260924 04:35:56.328135 2636 raft_consensus.cc:2781] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:56.328395 2636 raft_consensus.cc:686] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 LEADER]: Becoming Leader. State: Replica: 771c5c02ea42408d9241d2fc916dbcd0, State: Running, Role: LEADER
I20260924 04:35:56.328709 2636 consensus_queue.cc:237] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.332029 2122 catalog_manager.cc:5495] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } health_report { overall_health: HEALTHY } } }
W20260924 04:35:56.351512 2466 tablet.cc:2369] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:35:56.381474 2326 tablet.cc:2369] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:35:56 dist-test-slave-jlnt krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:56.405205 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.394758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35623 (local address 127.0.0.1:57408)
0924 04:35:56.395054 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.395066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.395203 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.395445 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.395447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.396111 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:35:56.396246 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.396253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.397203 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.397209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.397840 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.397848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.398302 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.398305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.398426 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.399551 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:35:56.399563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:35:56.401281 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:35:56.403840 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:56.403847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:56.403852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:56.404073 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:56.404306 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:35:56.404308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:35:56.404309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:35:56.404382 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:35:56.404712 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:35:56.404716 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:35:56.404818 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.404907 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.405011 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":23,"thread_start_us":113,"threads_started":1}
I20260924 04:35:56.405350 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.394807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.0.98.62:35623)
0924 04:35:56.394950 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.394956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.395140 (+ 184us) server_negotiation.cc:407] Connection header received
0924 04:35:56.395296 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.395300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.395343 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.395429 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:56.396395 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.397073 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.397953 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.398195 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.398303 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.401406 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:56.401410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:56.401412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:56.401434 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:56.403750 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:56.404158 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:56.404160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:56.404161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:56.404231 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:56.404452 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:56.404454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:56.404456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:56.404627 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:56.404699 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.404995 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.405099 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":42}
I20260924 04:35:56.409507 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57408:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:35:56.410142 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:35:56.414719 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 822280d96d4d4602bec31cf04a86dee3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a9eb980fba94b4aac2751748eae1274]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:35:56.414732 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 822280d96d4d4602bec31cf04a86dee3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a9eb980fba94b4aac2751748eae1274]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:35:56.414732 2521 tablet_service.cc:1467] Processing CreateTablet for tablet 822280d96d4d4602bec31cf04a86dee3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a9eb980fba94b4aac2751748eae1274]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:35:56.415383 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822280d96d4d4602bec31cf04a86dee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:56.415380 2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822280d96d4d4602bec31cf04a86dee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:56.415378 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822280d96d4d4602bec31cf04a86dee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:56.417765 2628 tablet_bootstrap.cc:492] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: Bootstrap starting.
I20260924 04:35:56.418218 2629 tablet_bootstrap.cc:492] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: Bootstrap starting.
I20260924 04:35:56.418747 2628 tablet_bootstrap.cc:654] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:56.419034 2629 tablet_bootstrap.cc:654] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:56.419680 2630 tablet_bootstrap.cc:492] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: Bootstrap starting.
I20260924 04:35:56.419874 2628 tablet_bootstrap.cc:492] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: No bootstrap required, opened a new log
I20260924 04:35:56.419958 2628 ts_tablet_manager.cc:1397] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:35:56.420231 2629 tablet_bootstrap.cc:492] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: No bootstrap required, opened a new log
I20260924 04:35:56.420331 2629 ts_tablet_manager.cc:1397] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:35:56.420451 2628 raft_consensus.cc:348] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.420593 2628 raft_consensus.cc:374] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:56.420624 2630 tablet_bootstrap.cc:654] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:56.420632 2628 raft_consensus.cc:729] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06143ea5f11404db273399e823ec71e, State: Initialized, Role: FOLLOWER
I20260924 04:35:56.420768 2628 consensus_queue.cc:260] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.420785 2629 raft_consensus.cc:348] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.420889 2629 raft_consensus.cc:374] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:56.420926 2629 raft_consensus.cc:729] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7358f32ee76344fa896aa0db09c0aac6, State: Initialized, Role: FOLLOWER
I20260924 04:35:56.421065 2628 ts_tablet_manager.cc:1428] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:56.421051 2629 consensus_queue.cc:260] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.421332 2629 ts_tablet_manager.cc:1428] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:56.421303 2634 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.421439 2634 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:56.422050 2630 tablet_bootstrap.cc:492] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: No bootstrap required, opened a new log
I20260924 04:35:56.422132 2630 ts_tablet_manager.cc:1397] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 04:35:56.422485 2635 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.422621 2635 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:56.422572 2630 raft_consensus.cc:348] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.422677 2630 raft_consensus.cc:374] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:56.422711 2630 raft_consensus.cc:729] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771c5c02ea42408d9241d2fc916dbcd0, State: Initialized, Role: FOLLOWER
I20260924 04:35:56.422851 2630 consensus_queue.cc:260] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.423118 2630 ts_tablet_manager.cc:1428] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:35:56.423074 2636 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.423185 2636 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:56.427100 2636 raft_consensus.cc:482] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:35:56.427179 2636 raft_consensus.cc:504] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.427546 2636 leader_election.cc:288] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721), f06143ea5f11404db273399e823ec71e (127.0.98.2:42955)
I20260924 04:35:56.427861 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822280d96d4d4602bec31cf04a86dee3" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358f32ee76344fa896aa0db09c0aac6" is_pre_election: true
I20260924 04:35:56.427964 2279 raft_consensus.cc:2454] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 0.
I20260924 04:35:56.428181 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822280d96d4d4602bec31cf04a86dee3" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06143ea5f11404db273399e823ec71e" is_pre_election: true
I20260924 04:35:56.428243 2492 leader_election.cc:302] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7358f32ee76344fa896aa0db09c0aac6, 771c5c02ea42408d9241d2fc916dbcd0; no voters:
I20260924 04:35:56.428299 2419 raft_consensus.cc:2454] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 0.
I20260924 04:35:56.428390 2636 raft_consensus.cc:2781] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:35:56.428463 2636 raft_consensus.cc:482] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:35:56.428516 2636 raft_consensus.cc:3037] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:56.429519 2636 raft_consensus.cc:504] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.429870 2636 leader_election.cc:288] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 election: Requested vote from peers 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721), f06143ea5f11404db273399e823ec71e (127.0.98.2:42955)
I20260924 04:35:56.430164 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822280d96d4d4602bec31cf04a86dee3" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358f32ee76344fa896aa0db09c0aac6"
I20260924 04:35:56.430192 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822280d96d4d4602bec31cf04a86dee3" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06143ea5f11404db273399e823ec71e"
I20260924 04:35:56.430269 2279 raft_consensus.cc:3037] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:56.430325 2419 raft_consensus.cc:3037] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:56.431097 2419 raft_consensus.cc:2454] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 1.
I20260924 04:35:56.431113 2279 raft_consensus.cc:2454] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 1.
I20260924 04:35:56.431375 2490 leader_election.cc:302] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771c5c02ea42408d9241d2fc916dbcd0, f06143ea5f11404db273399e823ec71e; no voters:
I20260924 04:35:56.431578 2636 raft_consensus.cc:2781] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:56.431667 2636 raft_consensus.cc:686] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 LEADER]: Becoming Leader. State: Replica: 771c5c02ea42408d9241d2fc916dbcd0, State: Running, Role: LEADER
I20260924 04:35:56.431780 2636 consensus_queue.cc:237] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.432451 2638 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: TxnStatusTablet state changed. Reason: New leader 771c5c02ea42408d9241d2fc916dbcd0. Latest consensus state: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.432554 2638 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:35:56.432912 2650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:35:56.433194 2650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:35:56.433328 2122 catalog_manager.cc:5495] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } health_report { overall_health: HEALTHY } } }
I20260924 04:35:56.446547 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.442753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36077 (local address 127.0.0.1:60656)
0924 04:35:56.443021 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.443052 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.443185 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.443495 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.443498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.443506 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:35:56.443609 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.443616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.444323 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.444328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.444934 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.444942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.445485 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.445489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.445580 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.445625 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.446258 (+ 633us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.446272 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.446334 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.446421 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260924 04:35:56.446578 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.442851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.0.98.3:36077)
0924 04:35:56.442973 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.442978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.443122 (+ 144us) server_negotiation.cc:407] Connection header received
0924 04:35:56.443322 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.443325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.443386 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.443485 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:56.443719 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.444219 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.445060 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.445377 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.445485 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.445774 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.446152 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.446271 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.446360 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.446408 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:35:56.451315 2419 raft_consensus.cc:1261] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Refusing update from remote peer 771c5c02ea42408d9241d2fc916dbcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:56.451352 2279 raft_consensus.cc:1261] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Refusing update from remote peer 771c5c02ea42408d9241d2fc916dbcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:56.451982 2636 consensus_queue.cc:1035] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:56.452162 2638 consensus_queue.cc:1035] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:56.455375 2634 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: TxnStatusTablet state changed. Reason: New leader 771c5c02ea42408d9241d2fc916dbcd0. Latest consensus state: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.455493 2634 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:56.456220 2635 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: TxnStatusTablet state changed. Reason: New leader 771c5c02ea42408d9241d2fc916dbcd0. Latest consensus state: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.456357 2635 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:56.459741 2636 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.459882 2636 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:35:56.460093 2635 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.460094 2634 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.460191 2634 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P f06143ea5f11404db273399e823ec71e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:56.460191 2635 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P 7358f32ee76344fa896aa0db09c0aac6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:35:56.461695 2654 mvcc.cc:205] Tried to move back new op lower bound from 7332759783217983488 to 7332759783145865216. Current Snapshot: MvccSnapshot[applied={T|T < 7332759783217983488}]
I20260924 04:35:56.461869 2655 mvcc.cc:205] Tried to move back new op lower bound from 7332759783217983488 to 7332759783145865216. Current Snapshot: MvccSnapshot[applied={T|T < 7332759783217983488}]
I20260924 04:35:56.463222 2636 tablet_replica.cc:439] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } } }
I20260924 04:35:56.463330 2636 tablet_replica.cc:442] T 822280d96d4d4602bec31cf04a86dee3 P 771c5c02ea42408d9241d2fc916dbcd0: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:35:56.472457 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.468910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.0.98.3:36077)
0924 04:35:56.469028 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.469032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.469068 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:35:56.469269 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.469272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.469321 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.469398 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:56.469674 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.470483 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.471328 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.471693 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.471788 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.472021 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.472099 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.472166 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.472249 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.472301 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:35:56.484252 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.480940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36077 (local address 127.0.98.3:47829)
0924 04:35:56.481070 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.481085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.481209 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.481454 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.481456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.481461 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:35:56.481548 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.481552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.482483 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.482485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.483092 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.483099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.483620 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.483622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.483688 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.483725 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.484004 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.484008 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.484067 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.484139 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260924 04:35:56.484275 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.480998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47829 (local address 127.0.98.3:36077)
0924 04:35:56.481097 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.481101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.481118 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:35:56.481320 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.481323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.481369 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.481442 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:56.481639 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.482378 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.483195 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.483519 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.483619 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.483855 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.483927 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.484003 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.484080 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.484147 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":39}
I20260924 04:35:56.490743 2419 raft_consensus.cc:1261] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Refusing update from remote peer 771c5c02ea42408d9241d2fc916dbcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:56.490744 2279 raft_consensus.cc:1261] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Refusing update from remote peer 771c5c02ea42408d9241d2fc916dbcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:56.491108 2636 consensus_queue.cc:1035] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:56.491276 2658 consensus_queue.cc:1035] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:56.496053 2653 mvcc.cc:205] Tried to move back new op lower bound from 7332759783383085056 to 7332759782725300224. Current Snapshot: MvccSnapshot[applied={T|T < 7332759783383085056}]
I20260924 04:35:56.532830 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:56.533021 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:56.535537 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: Sending DeleteTablet for 3 replicas of tablet e25f0e9fee994f88a04f5d407e0ac25e
I20260924 04:35:56.536273 2521 tablet_service.cc:1514] Processing DeleteTablet for tablet e25f0e9fee994f88a04f5d407e0ac25e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60642
I20260924 04:35:56.536325 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet e25f0e9fee994f88a04f5d407e0ac25e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50402
I20260924 04:35:56.536691 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet e25f0e9fee994f88a04f5d407e0ac25e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53610
I20260924 04:35:56.536914 2669 tablet_replica.cc:330] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: stopping tablet replica
I20260924 04:35:56.537190 2670 tablet_replica.cc:330] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: stopping tablet replica
I20260924 04:35:56.537304 2669 raft_consensus.cc:2229] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:35:56.537518 2669 raft_consensus.cc:2258] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:56.537674 2670 raft_consensus.cc:2229] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:35:56.537905 2670 raft_consensus.cc:2258] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:56.537829 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:35:56.538208 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:35:56.539003 2669 ts_tablet_manager.cc:1905] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:56.539511 2670 ts_tablet_manager.cc:1905] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:56.539865 2668 tablet_replica.cc:330] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: stopping tablet replica
I20260924 04:35:56.540150 2668 raft_consensus.cc:2229] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:35:56.541417 2668 raft_consensus.cc:2258] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:56.542693 2670 ts_tablet_manager.cc:1918] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:35:56.542693 2669 ts_tablet_manager.cc:1918] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:35:56.542786 2669 log.cc:1196] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/e25f0e9fee994f88a04f5d407e0ac25e
I20260924 04:35:56.542976 2670 log.cc:1196] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/e25f0e9fee994f88a04f5d407e0ac25e
I20260924 04:35:56.543205 2259 tablet_service.cc:1467] Processing CreateTablet for tablet a72c2f52ba6a46068e2e3f80e80ce60e (DEFAULT_TABLE table=test-table [id=24c658c9d17644a38f82e806e1226902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:56.543239 2521 tablet_service.cc:1467] Processing CreateTablet for tablet a72c2f52ba6a46068e2e3f80e80ce60e (DEFAULT_TABLE table=test-table [id=24c658c9d17644a38f82e806e1226902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:56.543459 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72c2f52ba6a46068e2e3f80e80ce60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:56.543476 2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72c2f52ba6a46068e2e3f80e80ce60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:56.543563 2669 ts_tablet_manager.cc:1939] T e25f0e9fee994f88a04f5d407e0ac25e P 7358f32ee76344fa896aa0db09c0aac6: Deleting consensus metadata
I20260924 04:35:56.544699 2108 catalog_manager.cc:4841] TS 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721): tablet e25f0e9fee994f88a04f5d407e0ac25e (table test-table [id=529d22bf9304494f8f6d07df342235ef]) successfully deleted
I20260924 04:35:56.543222 2394 tablet_service.cc:1467] Processing CreateTablet for tablet a72c2f52ba6a46068e2e3f80e80ce60e (DEFAULT_TABLE table=test-table [id=24c658c9d17644a38f82e806e1226902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:56.545526 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72c2f52ba6a46068e2e3f80e80ce60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:56.545526 2629 tablet_bootstrap.cc:492] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: Bootstrap starting.
I20260924 04:35:56.546422 2629 tablet_bootstrap.cc:654] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:56.546965 2668 ts_tablet_manager.cc:1905] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:56.547019 2630 tablet_bootstrap.cc:492] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: Bootstrap starting.
I20260924 04:35:56.547644 2628 tablet_bootstrap.cc:492] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: Bootstrap starting.
I20260924 04:35:56.547966 2630 tablet_bootstrap.cc:654] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:56.548187 2670 ts_tablet_manager.cc:1939] T e25f0e9fee994f88a04f5d407e0ac25e P f06143ea5f11404db273399e823ec71e: Deleting consensus metadata
I20260924 04:35:56.548540 2628 tablet_bootstrap.cc:654] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:56.549432 2630 tablet_bootstrap.cc:492] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: No bootstrap required, opened a new log
I20260924 04:35:56.549548 2630 ts_tablet_manager.cc:1397] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 04:35:56.549727 2106 catalog_manager.cc:4841] TS f06143ea5f11404db273399e823ec71e (127.0.98.2:42955): tablet e25f0e9fee994f88a04f5d407e0ac25e (table test-table [id=529d22bf9304494f8f6d07df342235ef]) successfully deleted
I20260924 04:35:56.550001 2630 raft_consensus.cc:348] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.550117 2630 raft_consensus.cc:374] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:56.550155 2630 raft_consensus.cc:729] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771c5c02ea42408d9241d2fc916dbcd0, State: Initialized, Role: FOLLOWER
I20260924 04:35:56.550278 2630 consensus_queue.cc:260] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.550559 2630 ts_tablet_manager.cc:1428] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:35:56.551702 2629 tablet_bootstrap.cc:492] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: No bootstrap required, opened a new log
I20260924 04:35:56.551811 2629 ts_tablet_manager.cc:1397] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 04:35:56.552284 2629 raft_consensus.cc:348] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.552405 2629 raft_consensus.cc:374] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:56.552444 2629 raft_consensus.cc:729] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7358f32ee76344fa896aa0db09c0aac6, State: Initialized, Role: FOLLOWER
I20260924 04:35:56.552593 2629 consensus_queue.cc:260] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.552846 2629 ts_tablet_manager.cc:1428] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:56.553516 2668 ts_tablet_manager.cc:1918] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:35:56.554445 2628 tablet_bootstrap.cc:492] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: No bootstrap required, opened a new log
I20260924 04:35:56.554533 2628 ts_tablet_manager.cc:1397] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 04:35:56.554939 2628 raft_consensus.cc:348] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.555060 2628 raft_consensus.cc:374] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:56.555090 2628 raft_consensus.cc:729] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06143ea5f11404db273399e823ec71e, State: Initialized, Role: FOLLOWER
I20260924 04:35:56.555200 2628 consensus_queue.cc:260] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.555437 2628 ts_tablet_manager.cc:1428] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:56.553987 2668 log.cc:1196] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/e25f0e9fee994f88a04f5d407e0ac25e
I20260924 04:35:56.556824 2668 ts_tablet_manager.cc:1939] T e25f0e9fee994f88a04f5d407e0ac25e P 771c5c02ea42408d9241d2fc916dbcd0: Deleting consensus metadata
I20260924 04:35:56.558094 2106 catalog_manager.cc:4841] TS 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3:36077): tablet e25f0e9fee994f88a04f5d407e0ac25e (table test-table [id=529d22bf9304494f8f6d07df342235ef]) successfully deleted
I20260924 04:35:56.762454 2635 raft_consensus.cc:482] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:35:56.762621 2635 raft_consensus.cc:504] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.763777 2635 leader_election.cc:288] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f06143ea5f11404db273399e823ec71e (127.0.98.2:42955), 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3:36077)
I20260924 04:35:56.767855 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.764602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59909 (local address 127.0.98.3:36077)
0924 04:35:56.764736 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.764743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.764762 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:35:56.764914 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.764917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.764964 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.765051 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:56.765304 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.765790 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.766894 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.767292 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.767554 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.767637 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.767674 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.767731 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:35:56.767866 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.764005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42955 (local address 127.0.98.1:58873)
0924 04:35:56.764251 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.764269 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.764387 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.764922 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.764925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.764945 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:56.765047 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.765053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.765842 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.765847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.766777 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.766786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.767428 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.767432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.767520 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.767551 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.767666 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.767735 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":53,"thread_start_us":107,"threads_started":1}
I20260924 04:35:56.767890 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.764391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36077 (local address 127.0.98.1:59909)
0924 04:35:56.764703 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.764716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.764803 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.765055 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.765058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.765075 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:56.765195 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.765207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.765893 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.765899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.766775 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.766784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.767456 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.767459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.767566 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.767586 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.767653 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.767729 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":36,"thread_start_us":105,"threads_started":1}
I20260924 04:35:56.768260 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.764145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58873 (local address 127.0.98.2:42955)
0924 04:35:56.764280 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.764283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.764329 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:35:56.764569 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.764573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.764631 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.764908 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:56.765213 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.765728 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.766900 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.767306 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.767426 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.767491 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.768062 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.768158 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:35:56.768496 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72c2f52ba6a46068e2e3f80e80ce60e" candidate_uuid: "7358f32ee76344fa896aa0db09c0aac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771c5c02ea42408d9241d2fc916dbcd0" is_pre_election: true
I20260924 04:35:56.768734 2559 raft_consensus.cc:2454] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7358f32ee76344fa896aa0db09c0aac6 in term 0.
I20260924 04:35:56.768772 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72c2f52ba6a46068e2e3f80e80ce60e" candidate_uuid: "7358f32ee76344fa896aa0db09c0aac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06143ea5f11404db273399e823ec71e" is_pre_election: true
I20260924 04:35:56.768888 2419 raft_consensus.cc:2454] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7358f32ee76344fa896aa0db09c0aac6 in term 0.
I20260924 04:35:56.769232 2210 leader_election.cc:302] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7358f32ee76344fa896aa0db09c0aac6, 771c5c02ea42408d9241d2fc916dbcd0; no voters:
I20260924 04:35:56.769487 2635 raft_consensus.cc:2781] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:35:56.769551 2635 raft_consensus.cc:482] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:35:56.769589 2635 raft_consensus.cc:3037] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:56.770987 2635 raft_consensus.cc:504] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.771371 2635 leader_election.cc:288] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [CANDIDATE]: Term 1 election: Requested vote from peers f06143ea5f11404db273399e823ec71e (127.0.98.2:42955), 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3:36077)
I20260924 04:35:56.771742 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72c2f52ba6a46068e2e3f80e80ce60e" candidate_uuid: "7358f32ee76344fa896aa0db09c0aac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771c5c02ea42408d9241d2fc916dbcd0"
I20260924 04:35:56.771744 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72c2f52ba6a46068e2e3f80e80ce60e" candidate_uuid: "7358f32ee76344fa896aa0db09c0aac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06143ea5f11404db273399e823ec71e"
I20260924 04:35:56.771875 2559 raft_consensus.cc:3037] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:56.771890 2419 raft_consensus.cc:3037] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:56.772840 2559 raft_consensus.cc:2454] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7358f32ee76344fa896aa0db09c0aac6 in term 1.
I20260924 04:35:56.772851 2419 raft_consensus.cc:2454] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7358f32ee76344fa896aa0db09c0aac6 in term 1.
I20260924 04:35:56.773124 2210 leader_election.cc:302] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7358f32ee76344fa896aa0db09c0aac6, f06143ea5f11404db273399e823ec71e; no voters:
I20260924 04:35:56.773314 2635 raft_consensus.cc:2781] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:56.773509 2635 raft_consensus.cc:686] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 1 LEADER]: Becoming Leader. State: Replica: 7358f32ee76344fa896aa0db09c0aac6, State: Running, Role: LEADER
I20260924 04:35:56.773765 2635 consensus_queue.cc:237] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:56.775846 2120 catalog_manager.cc:5495] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7358f32ee76344fa896aa0db09c0aac6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } health_report { overall_health: UNKNOWN } } }
I20260924 04:35:56.791352 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.787790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.0.98.1:34721)
0924 04:35:56.787892 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.787897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.787968 (+ 71us) server_negotiation.cc:407] Connection header received
0924 04:35:56.788122 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.788125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.788182 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.788287 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:56.788520 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.789054 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.789945 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.790175 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.790286 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.790548 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.790983 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.791110 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.791167 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.791217 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:35:56.800599 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.797632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49279 (local address 127.0.98.3:36077)
0924 04:35:56.797780 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.797784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.797948 (+ 164us) server_negotiation.cc:407] Connection header received
0924 04:35:56.798073 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.798077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.798136 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.798197 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:56.798421 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.798929 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.799750 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.799945 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.800015 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.800267 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.800325 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.800374 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.800460 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.800504 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:35:56.800616 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.797576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36077 (local address 127.0.98.1:49279)
0924 04:35:56.797876 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.797887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.797967 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.798221 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.798223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.798229 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:35:56.798318 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.798322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.799044 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.799047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.799651 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.799659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.800048 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.800051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.800124 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.800163 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.800393 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.800397 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.800444 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.800500 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":84,"threads_started":1}
I20260924 04:35:56.806535 2419 raft_consensus.cc:1261] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Refusing update from remote peer 7358f32ee76344fa896aa0db09c0aac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:56.806727 2559 raft_consensus.cc:1261] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Refusing update from remote peer 7358f32ee76344fa896aa0db09c0aac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:56.807017 2676 consensus_queue.cc:1035] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:56.807178 2635 consensus_queue.cc:1035] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:56.824622 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.821485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46081 (local address 127.0.98.1:34721)
0924 04:35:56.821879 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:56.821883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:56.821892 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:35:56.821924 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:56.821925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:56.821988 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:56.822068 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:56.822249 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.822894 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.823648 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.823869 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.823998 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.824220 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.824289 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.824367 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:56.824457 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:35:56.824500 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260924 04:35:56.824630 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:56.821446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34721 (local address 127.0.98.3:46081)
0924 04:35:56.821619 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:56.821632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:56.821724 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:56.822090 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:56.822092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:56.822099 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:35:56.822177 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.822181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.822988 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.822990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.823537 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:56.823545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:56.823981 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:56.823984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:56.824070 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:56.824106 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:35:56.824370 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:35:56.824378 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 04:35:56.824438 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:35:56.824522 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 04:35:56.825361 2654 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:35:58.378742 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:58.378916 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:58.381043 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: Sending DeleteTablet for 3 replicas of tablet a72c2f52ba6a46068e2e3f80e80ce60e
I20260924 04:35:58.381620 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet a72c2f52ba6a46068e2e3f80e80ce60e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50402
I20260924 04:35:58.381628 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet a72c2f52ba6a46068e2e3f80e80ce60e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53610
I20260924 04:35:58.381675 2521 tablet_service.cc:1514] Processing DeleteTablet for tablet a72c2f52ba6a46068e2e3f80e80ce60e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60642
I20260924 04:35:58.381990 2708 tablet_replica.cc:330] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: stopping tablet replica
I20260924 04:35:58.382193 2708 raft_consensus.cc:2229] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:35:58.382321 2708 raft_consensus.cc:2258] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:58.382869 2708 ts_tablet_manager.cc:1905] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:58.385280 2709 tablet_replica.cc:330] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: stopping tablet replica
I20260924 04:35:58.385460 2709 raft_consensus.cc:2229] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:35:58.385578 2709 raft_consensus.cc:2258] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:58.385653 2708 ts_tablet_manager.cc:1918] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:35:58.385728 2708 log.cc:1196] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/a72c2f52ba6a46068e2e3f80e80ce60e
I20260924 04:35:58.385771 2707 tablet_replica.cc:330] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: stopping tablet replica
I20260924 04:35:58.385908 2707 raft_consensus.cc:2229] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:35:58.386015 2708 ts_tablet_manager.cc:1939] T a72c2f52ba6a46068e2e3f80e80ce60e P f06143ea5f11404db273399e823ec71e: Deleting consensus metadata
I20260924 04:35:58.386241 2707 raft_consensus.cc:2258] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:58.386435 2709 ts_tablet_manager.cc:1905] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:58.386657 2106 catalog_manager.cc:4841] TS f06143ea5f11404db273399e823ec71e (127.0.98.2:42955): tablet a72c2f52ba6a46068e2e3f80e80ce60e (table test-table [id=24c658c9d17644a38f82e806e1226902]) successfully deleted
I20260924 04:35:58.386873 2707 ts_tablet_manager.cc:1905] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:58.389045 2707 ts_tablet_manager.cc:1918] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:35:58.389137 2707 log.cc:1196] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/a72c2f52ba6a46068e2e3f80e80ce60e
I20260924 04:35:58.389304 2709 ts_tablet_manager.cc:1918] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:35:58.389391 2709 log.cc:1196] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/a72c2f52ba6a46068e2e3f80e80ce60e
I20260924 04:35:58.389485 2707 ts_tablet_manager.cc:1939] T a72c2f52ba6a46068e2e3f80e80ce60e P 7358f32ee76344fa896aa0db09c0aac6: Deleting consensus metadata
I20260924 04:35:58.389699 2709 ts_tablet_manager.cc:1939] T a72c2f52ba6a46068e2e3f80e80ce60e P 771c5c02ea42408d9241d2fc916dbcd0: Deleting consensus metadata
I20260924 04:35:58.390126 2108 catalog_manager.cc:4841] TS 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721): tablet a72c2f52ba6a46068e2e3f80e80ce60e (table test-table [id=24c658c9d17644a38f82e806e1226902]) successfully deleted
I20260924 04:35:58.390334 2106 catalog_manager.cc:4841] TS 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3:36077): tablet a72c2f52ba6a46068e2e3f80e80ce60e (table test-table [id=24c658c9d17644a38f82e806e1226902]) successfully deleted
Sep 24 04:35:58 dist-test-slave-jlnt krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:35:58 dist-test-slave-jlnt krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224558, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:35:58.410405 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.400101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.0.98.62:35623)
0924 04:35:58.400404 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:58.400414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:58.400428 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:35:58.400560 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:58.400563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:58.400639 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:58.400737 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:35:58.401461 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.402008 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.402895 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.403125 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.403234 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.406465 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:35:58.406469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:35:58.406471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:35:58.406493 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:35:58.408815 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:58.409352 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:58.409354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:58.409355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:58.409428 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:35:58.409685 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:35:58.409687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:35:58.409688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:35:58.409837 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:35:58.409927 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:58.410097 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 04:35:58.410192 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
I20260924 04:35:58.414049 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:35:58.414426 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:35:58.418910 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 61a1c5d4975047f4911d962f3904ff97 (DEFAULT_TABLE table=test-table [id=6c8355cb3a384bcdb20e91dfa27d21eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:58.418944 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 61a1c5d4975047f4911d962f3904ff97 (DEFAULT_TABLE table=test-table [id=6c8355cb3a384bcdb20e91dfa27d21eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:58.418980 2521 tablet_service.cc:1467] Processing CreateTablet for tablet 61a1c5d4975047f4911d962f3904ff97 (DEFAULT_TABLE table=test-table [id=6c8355cb3a384bcdb20e91dfa27d21eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:58.419209 2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a1c5d4975047f4911d962f3904ff97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:58.419211 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a1c5d4975047f4911d962f3904ff97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:58.419224 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a1c5d4975047f4911d962f3904ff97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:58.421798 2719 tablet_bootstrap.cc:492] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: Bootstrap starting.
I20260924 04:35:58.422022 2720 tablet_bootstrap.cc:492] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: Bootstrap starting.
I20260924 04:35:58.422188 2718 tablet_bootstrap.cc:492] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: Bootstrap starting.
I20260924 04:35:58.422708 2719 tablet_bootstrap.cc:654] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:58.422719 2720 tablet_bootstrap.cc:654] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:58.423123 2718 tablet_bootstrap.cc:654] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:58.423758 2720 tablet_bootstrap.cc:492] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: No bootstrap required, opened a new log
I20260924 04:35:58.423859 2720 ts_tablet_manager.cc:1397] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:35:58.424327 2720 raft_consensus.cc:348] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.424464 2720 raft_consensus.cc:374] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:58.424512 2720 raft_consensus.cc:729] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771c5c02ea42408d9241d2fc916dbcd0, State: Initialized, Role: FOLLOWER
I20260924 04:35:58.424674 2720 consensus_queue.cc:260] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.424960 2719 tablet_bootstrap.cc:492] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: No bootstrap required, opened a new log
I20260924 04:35:58.425055 2719 ts_tablet_manager.cc:1397] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:35:58.425392 2720 ts_tablet_manager.cc:1428] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:58.425556 2719 raft_consensus.cc:348] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.425655 2719 raft_consensus.cc:374] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:58.425673 2718 tablet_bootstrap.cc:492] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: No bootstrap required, opened a new log
I20260924 04:35:58.425693 2719 raft_consensus.cc:729] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7358f32ee76344fa896aa0db09c0aac6, State: Initialized, Role: FOLLOWER
I20260924 04:35:58.425745 2718 ts_tablet_manager.cc:1397] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:35:58.425840 2719 consensus_queue.cc:260] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.426172 2718 raft_consensus.cc:348] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.426283 2718 raft_consensus.cc:374] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:58.426326 2718 raft_consensus.cc:729] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06143ea5f11404db273399e823ec71e, State: Initialized, Role: FOLLOWER
I20260924 04:35:58.426440 2718 consensus_queue.cc:260] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.426510 2719 ts_tablet_manager.cc:1428] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:58.426789 2718 ts_tablet_manager.cc:1428] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:58.440939 2702 raft_consensus.cc:482] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:35:58.441043 2702 raft_consensus.cc:504] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.441445 2702 leader_election.cc:288] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721), f06143ea5f11404db273399e823ec71e (127.0.98.2:42955)
I20260924 04:35:58.441864 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a1c5d4975047f4911d962f3904ff97" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06143ea5f11404db273399e823ec71e" is_pre_election: true
I20260924 04:35:58.441864 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a1c5d4975047f4911d962f3904ff97" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358f32ee76344fa896aa0db09c0aac6" is_pre_election: true
I20260924 04:35:58.441988 2279 raft_consensus.cc:2454] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 0.
I20260924 04:35:58.441988 2419 raft_consensus.cc:2454] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 0.
I20260924 04:35:58.442278 2490 leader_election.cc:302] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771c5c02ea42408d9241d2fc916dbcd0, f06143ea5f11404db273399e823ec71e; no voters:
I20260924 04:35:58.442467 2702 raft_consensus.cc:2781] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:35:58.442524 2702 raft_consensus.cc:482] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:35:58.442561 2702 raft_consensus.cc:3037] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:58.443364 2702 raft_consensus.cc:504] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.443655 2702 leader_election.cc:288] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 election: Requested vote from peers 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721), f06143ea5f11404db273399e823ec71e (127.0.98.2:42955)
I20260924 04:35:58.443912 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a1c5d4975047f4911d962f3904ff97" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358f32ee76344fa896aa0db09c0aac6"
I20260924 04:35:58.443998 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a1c5d4975047f4911d962f3904ff97" candidate_uuid: "771c5c02ea42408d9241d2fc916dbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06143ea5f11404db273399e823ec71e"
I20260924 04:35:58.444063 2279 raft_consensus.cc:3037] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:58.444124 2419 raft_consensus.cc:3037] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:58.444865 2279 raft_consensus.cc:2454] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 1.
I20260924 04:35:58.444932 2419 raft_consensus.cc:2454] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771c5c02ea42408d9241d2fc916dbcd0 in term 1.
I20260924 04:35:58.445183 2492 leader_election.cc:302] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7358f32ee76344fa896aa0db09c0aac6, 771c5c02ea42408d9241d2fc916dbcd0; no voters:
I20260924 04:35:58.445362 2702 raft_consensus.cc:2781] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:58.445473 2702 raft_consensus.cc:686] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 LEADER]: Becoming Leader. State: Replica: 771c5c02ea42408d9241d2fc916dbcd0, State: Running, Role: LEADER
I20260924 04:35:58.445628 2702 consensus_queue.cc:237] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.447008 2120 catalog_manager.cc:5495] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "771c5c02ea42408d9241d2fc916dbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } health_report { overall_health: HEALTHY } } }
I20260924 04:35:58.460414 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.457180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.98.3:36077)
0924 04:35:58.457420 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:58.457427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:58.457454 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:35:58.457489 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:58.457491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:58.457527 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:58.457636 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:58.457869 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.458378 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.459377 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.459594 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.459742 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.459928 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:58.459990 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:58.460041 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:58.460150 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:35:58.460193 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
I20260924 04:35:58.466210 2279 raft_consensus.cc:1261] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Refusing update from remote peer 771c5c02ea42408d9241d2fc916dbcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:58.466223 2419 raft_consensus.cc:1261] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Refusing update from remote peer 771c5c02ea42408d9241d2fc916dbcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:58.466524 2702 consensus_queue.cc:1035] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:58.466701 2685 consensus_queue.cc:1035] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:58.496315 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:58.496454 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:35:58.498332 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: Sending DeleteTablet for 3 replicas of tablet 61a1c5d4975047f4911d962f3904ff97
I20260924 04:35:58.498864 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a1c5d4975047f4911d962f3904ff97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50402
I20260924 04:35:58.498865 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a1c5d4975047f4911d962f3904ff97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60642
I20260924 04:35:58.498955 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a1c5d4975047f4911d962f3904ff97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:35:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53610
I20260924 04:35:58.499053 2707 tablet_replica.cc:330] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: stopping tablet replica
I20260924 04:35:58.499130 2708 tablet_replica.cc:330] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: stopping tablet replica
I20260924 04:35:58.499177 2707 raft_consensus.cc:2229] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:35:58.499238 2708 raft_consensus.cc:2229] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:35:58.499284 2707 raft_consensus.cc:2258] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:58.499354 2708 raft_consensus.cc:2258] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:58.499441 2709 tablet_replica.cc:330] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: stopping tablet replica
I20260924 04:35:58.499563 2709 raft_consensus.cc:2229] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:35:58.499725 2709 raft_consensus.cc:2258] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:35:58.500028 2708 ts_tablet_manager.cc:1905] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:58.500368 2707 ts_tablet_manager.cc:1905] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:58.500620 2709 ts_tablet_manager.cc:1905] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:35:58.500622 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:35:58.501205 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:35:58.502615 2708 ts_tablet_manager.cc:1918] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:35:58.502769 2708 log.cc:1196] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/61a1c5d4975047f4911d962f3904ff97
I20260924 04:35:58.502856 2707 ts_tablet_manager.cc:1918] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:35:58.502940 2707 log.cc:1196] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/61a1c5d4975047f4911d962f3904ff97
I20260924 04:35:58.503046 2708 ts_tablet_manager.cc:1939] T 61a1c5d4975047f4911d962f3904ff97 P f06143ea5f11404db273399e823ec71e: Deleting consensus metadata
I20260924 04:35:58.503088 2709 ts_tablet_manager.cc:1918] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:35:58.503166 2709 log.cc:1196] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/61a1c5d4975047f4911d962f3904ff97
I20260924 04:35:58.503226 2707 ts_tablet_manager.cc:1939] T 61a1c5d4975047f4911d962f3904ff97 P 7358f32ee76344fa896aa0db09c0aac6: Deleting consensus metadata
I20260924 04:35:58.503432 2709 ts_tablet_manager.cc:1939] T 61a1c5d4975047f4911d962f3904ff97 P 771c5c02ea42408d9241d2fc916dbcd0: Deleting consensus metadata
I20260924 04:35:58.503638 2106 catalog_manager.cc:4841] TS f06143ea5f11404db273399e823ec71e (127.0.98.2:42955): tablet 61a1c5d4975047f4911d962f3904ff97 (table test-table [id=6c8355cb3a384bcdb20e91dfa27d21eb]) successfully deleted
I20260924 04:35:58.503813 2108 catalog_manager.cc:4841] TS 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721): tablet 61a1c5d4975047f4911d962f3904ff97 (table test-table [id=6c8355cb3a384bcdb20e91dfa27d21eb]) successfully deleted
I20260924 04:35:58.503979 2106 catalog_manager.cc:4841] TS 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3:36077): tablet 61a1c5d4975047f4911d962f3904ff97 (table test-table [id=6c8355cb3a384bcdb20e91dfa27d21eb]) successfully deleted
I20260924 04:35:58.507157 2394 tablet_service.cc:1467] Processing CreateTablet for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4 (DEFAULT_TABLE table=test-table [id=da51bc5c5dcf472b80315d9172d25086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:58.507154 2259 tablet_service.cc:1467] Processing CreateTablet for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4 (DEFAULT_TABLE table=test-table [id=da51bc5c5dcf472b80315d9172d25086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:58.507198 2538 tablet_service.cc:1467] Processing CreateTablet for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4 (DEFAULT_TABLE table=test-table [id=da51bc5c5dcf472b80315d9172d25086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:35:58.507404 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:58.507404 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:58.507404 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:35:58.508952 2718 tablet_bootstrap.cc:492] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: Bootstrap starting.
I20260924 04:35:58.509509 2720 tablet_bootstrap.cc:492] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: Bootstrap starting.
I20260924 04:35:58.509550 2719 tablet_bootstrap.cc:492] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6: Bootstrap starting.
I20260924 04:35:58.509895 2718 tablet_bootstrap.cc:654] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:58.510277 2719 tablet_bootstrap.cc:654] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:58.510295 2720 tablet_bootstrap.cc:654] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: Neither blocks nor log segments found. Creating new log.
I20260924 04:35:58.510979 2718 tablet_bootstrap.cc:492] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: No bootstrap required, opened a new log
I20260924 04:35:58.511080 2718 ts_tablet_manager.cc:1397] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:35:58.511483 2718 raft_consensus.cc:348] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.511602 2718 raft_consensus.cc:374] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:58.511646 2718 raft_consensus.cc:729] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06143ea5f11404db273399e823ec71e, State: Initialized, Role: FOLLOWER
I20260924 04:35:58.511737 2719 tablet_bootstrap.cc:492] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6: No bootstrap required, opened a new log
I20260924 04:35:58.511832 2719 ts_tablet_manager.cc:1397] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:35:58.511776 2718 consensus_queue.cc:260] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.512065 2718 ts_tablet_manager.cc:1428] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:35:58.512200 2719 raft_consensus.cc:348] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.512312 2719 raft_consensus.cc:374] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:58.512351 2719 raft_consensus.cc:729] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7358f32ee76344fa896aa0db09c0aac6, State: Initialized, Role: FOLLOWER
I20260924 04:35:58.512482 2719 consensus_queue.cc:260] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.512745 2719 ts_tablet_manager.cc:1428] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:58.512830 2720 tablet_bootstrap.cc:492] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: No bootstrap required, opened a new log
I20260924 04:35:58.512943 2720 ts_tablet_manager.cc:1397] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:35:58.513365 2720 raft_consensus.cc:348] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.513476 2720 raft_consensus.cc:374] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:35:58.513509 2720 raft_consensus.cc:729] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771c5c02ea42408d9241d2fc916dbcd0, State: Initialized, Role: FOLLOWER
I20260924 04:35:58.513612 2720 consensus_queue.cc:260] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.513904 2720 ts_tablet_manager.cc:1428] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:35:58.578253 2724 raft_consensus.cc:482] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:35:58.578408 2724 raft_consensus.cc:504] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.579501 2724 leader_election.cc:288] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721), 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3:36077)
I20260924 04:35:58.583348 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.579776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36077 (local address 127.0.98.2:41371)
0924 04:35:58.580179 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:58.580193 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:58.580287 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:58.580558 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:58.580560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:58.580574 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:58.580703 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:58.580709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.581390 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.581394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:58.582270 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:58.582278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.582894 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.582897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:58.583009 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.583031 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:35:58.583110 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:35:58.583164 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":3,"thread_start_us":113,"threads_started":1}
I20260924 04:35:58.583346 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.579941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41371 (local address 127.0.98.3:36077)
0924 04:35:58.580070 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:58.580074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:58.580276 (+ 202us) server_negotiation.cc:407] Connection header received
0924 04:35:58.580403 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:58.580406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:58.580456 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:58.580534 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:58.580801 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.581292 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.582385 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.582778 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.582856 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.582905 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:58.583137 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 04:35:58.583229 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:35:58.583400 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.579662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34721 (local address 127.0.98.2:34157)
0924 04:35:58.579998 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:58.580014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:58.580125 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:58.580413 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:58.580416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:58.580432 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:35:58.580566 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:58.580572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.581460 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.581463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:58.582275 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:58.582283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.583003 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.583006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:58.583109 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.583132 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:35:58.583209 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:35:58.583270 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":43,"thread_start_us":88,"threads_started":1}
I20260924 04:35:58.583669 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.579779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34157 (local address 127.0.98.1:34721)
0924 04:35:58.580029 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:58.580044 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:58.580066 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:35:58.580243 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:58.580249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:58.580304 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:58.580391 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:35:58.580677 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.581349 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.582409 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.582890 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.582992 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.583050 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:58.583236 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 04:35:58.583304 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":66,"threads_started":1}
I20260924 04:35:58.583782 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc9ccfba57c4fd98e5d6a88ab4a26b4" candidate_uuid: "f06143ea5f11404db273399e823ec71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771c5c02ea42408d9241d2fc916dbcd0" is_pre_election: true
I20260924 04:35:58.583941 2559 raft_consensus.cc:2454] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f06143ea5f11404db273399e823ec71e in term 0.
I20260924 04:35:58.584009 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc9ccfba57c4fd98e5d6a88ab4a26b4" candidate_uuid: "f06143ea5f11404db273399e823ec71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358f32ee76344fa896aa0db09c0aac6" is_pre_election: true
I20260924 04:35:58.584131 2279 raft_consensus.cc:2454] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f06143ea5f11404db273399e823ec71e in term 0.
I20260924 04:35:58.584336 2350 leader_election.cc:302] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771c5c02ea42408d9241d2fc916dbcd0, f06143ea5f11404db273399e823ec71e; no voters:
I20260924 04:35:58.584589 2724 raft_consensus.cc:2781] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:35:58.584661 2724 raft_consensus.cc:482] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:35:58.584715 2724 raft_consensus.cc:3037] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:58.585747 2724 raft_consensus.cc:504] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.586066 2724 leader_election.cc:288] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [CANDIDATE]: Term 1 election: Requested vote from peers 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721), 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3:36077)
I20260924 04:35:58.586390 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc9ccfba57c4fd98e5d6a88ab4a26b4" candidate_uuid: "f06143ea5f11404db273399e823ec71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358f32ee76344fa896aa0db09c0aac6"
I20260924 04:35:58.586388 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc9ccfba57c4fd98e5d6a88ab4a26b4" candidate_uuid: "f06143ea5f11404db273399e823ec71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771c5c02ea42408d9241d2fc916dbcd0"
I20260924 04:35:58.586506 2559 raft_consensus.cc:3037] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:58.586544 2279 raft_consensus.cc:3037] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:35:58.587584 2559 raft_consensus.cc:2454] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f06143ea5f11404db273399e823ec71e in term 1.
I20260924 04:35:58.587584 2279 raft_consensus.cc:2454] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f06143ea5f11404db273399e823ec71e in term 1.
I20260924 04:35:58.587848 2350 leader_election.cc:302] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771c5c02ea42408d9241d2fc916dbcd0, f06143ea5f11404db273399e823ec71e; no voters:
I20260924 04:35:58.588016 2724 raft_consensus.cc:2781] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:35:58.588244 2724 raft_consensus.cc:686] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 1 LEADER]: Becoming Leader. State: Replica: f06143ea5f11404db273399e823ec71e, State: Running, Role: LEADER
I20260924 04:35:58.588487 2724 consensus_queue.cc:237] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } }
I20260924 04:35:58.590961 2122 catalog_manager.cc:5495] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e reported cstate change: term changed from 0 to 1, leader changed from <none> to f06143ea5f11404db273399e823ec71e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f06143ea5f11404db273399e823ec71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f06143ea5f11404db273399e823ec71e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 } health_report { overall_health: UNKNOWN } } }
I20260924 04:35:58.619390 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.615471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.0.98.2:42955)
0924 04:35:58.615724 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:58.615734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:58.615746 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:35:58.615808 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:58.615810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:58.615856 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:58.616392 (+ 536us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:58.616627 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.617157 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.617953 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.618199 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.618307 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.618560 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:58.618996 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:58.619123 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:58.619185 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 04:35:58.619234 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
I20260924 04:35:58.629361 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.625715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36007 (local address 127.0.98.3:36077)
0924 04:35:58.625843 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:58.625848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:58.626004 (+ 156us) server_negotiation.cc:407] Connection header received
0924 04:35:58.626163 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:58.626167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:58.626219 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:58.626297 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:58.626582 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.627082 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.628314 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.628528 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.628645 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.628929 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:58.628994 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:58.629072 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:58.629193 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:35:58.629238 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:35:58.629388 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.625577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36077 (local address 127.0.98.2:36007)
0924 04:35:58.625913 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:58.625929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:58.626049 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:58.626308 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:58.626311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:58.626318 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:35:58.626469 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:58.626474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.627184 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.627188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:58.628194 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:58.628202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.628649 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.628653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:58.628763 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.628798 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:35:58.629083 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:35:58.629091 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 04:35:58.629181 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:35:58.629256 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":127,"threads_started":1}
I20260924 04:35:58.634418 2559 raft_consensus.cc:1261] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Refusing update from remote peer f06143ea5f11404db273399e823ec71e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:58.634567 2279 raft_consensus.cc:1261] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6 [term 1 FOLLOWER]: Refusing update from remote peer f06143ea5f11404db273399e823ec71e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:35:58.635030 2738 consensus_queue.cc:1035] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "771c5c02ea42408d9241d2fc916dbcd0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:58.635491 2738 consensus_queue.cc:1035] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7358f32ee76344fa896aa0db09c0aac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:35:58.650725 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.647319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42955 (local address 127.0.98.3:35359)
0924 04:35:58.647619 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:35:58.647633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:35:58.647728 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:35:58.647920 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:35:58.647923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:35:58.647930 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:35:58.648024 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:58.648029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.648757 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.648760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:58.649605 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:35:58.649616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.650190 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.650193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:35:58.650250 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.650275 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:35:58.650520 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:35:58.650523 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:35:58.650565 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 04:35:58.650617 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260924 04:35:58.650748 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:35:58.647385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35359 (local address 127.0.98.2:42955)
0924 04:35:58.647478 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:35:58.647482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:35:58.647716 (+ 234us) server_negotiation.cc:407] Connection header received
0924 04:35:58.647799 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:35:58.647802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:35:58.647837 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:35:58.647894 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:35:58.648117 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.648619 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.649729 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:35:58.650081 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:35:58.650176 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:35:58.650380 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:35:58.650450 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:35:58.650518 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:35:58.650581 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 04:35:58.650626 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 04:35:58.651335 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:36:00.202885 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:00.203030 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:00.205338 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e2d389a39934923bdf8d0b3fce0afd1: Sending DeleteTablet for 3 replicas of tablet adc9ccfba57c4fd98e5d6a88ab4a26b4
I20260924 04:36:00.205845 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53610
I20260924 04:36:00.205832 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50402
I20260924 04:36:00.206110 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60642
I20260924 04:36:00.206169 2771 tablet_replica.cc:330] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 7358f32ee76344fa896aa0db09c0aac6: stopping tablet replica
I20260924 04:36:00.206276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 2194
I20260924 04:36:00.206996 2772 tablet_replica.cc:330] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: stopping tablet replica
I20260924 04:36:00.207134 2772 raft_consensus.cc:2229] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:00.207432 2772 raft_consensus.cc:2258] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:00.207788 2773 tablet_replica.cc:330] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: stopping tablet replica
I20260924 04:36:00.207919 2773 raft_consensus.cc:2229] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:00.208055 2773 raft_consensus.cc:2258] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:00.208127 2772 ts_tablet_manager.cc:1905] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:00.208591 2773 ts_tablet_manager.cc:1905] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:00.210668 2773 ts_tablet_manager.cc:1918] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:36:00.210760 2773 log.cc:1196] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/adc9ccfba57c4fd98e5d6a88ab4a26b4
I20260924 04:36:00.211050 2773 ts_tablet_manager.cc:1939] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P 771c5c02ea42408d9241d2fc916dbcd0: Deleting consensus metadata
I20260924 04:36:00.211444 2772 ts_tablet_manager.cc:1918] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:36:00.211529 2772 log.cc:1196] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/adc9ccfba57c4fd98e5d6a88ab4a26b4
I20260924 04:36:00.211838 2772 ts_tablet_manager.cc:1939] T adc9ccfba57c4fd98e5d6a88ab4a26b4 P f06143ea5f11404db273399e823ec71e: Deleting consensus metadata
I20260924 04:36:00.211825 2106 catalog_manager.cc:4841] TS 771c5c02ea42408d9241d2fc916dbcd0 (127.0.98.3:36077): tablet adc9ccfba57c4fd98e5d6a88ab4a26b4 (table test-table [id=da51bc5c5dcf472b80315d9172d25086]) successfully deleted
I20260924 04:36:00.212481 2106 catalog_manager.cc:4841] TS f06143ea5f11404db273399e823ec71e (127.0.98.2:42955): tablet adc9ccfba57c4fd98e5d6a88ab4a26b4 (table test-table [id=da51bc5c5dcf472b80315d9172d25086]) successfully deleted
W20260924 04:36:00.215008 2108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34721 (error 108)
I20260924 04:36:00.215710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 2334
I20260924 04:36:00.216686 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:00.215666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34721 (local address 127.0.0.1:38532)
0924 04:36:00.216471 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:00.216557 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":90,"threads_started":1}
W20260924 04:36:00.216787 2108 catalog_manager.cc:4584] TS 7358f32ee76344fa896aa0db09c0aac6 (127.0.98.1:34721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adc9ccfba57c4fd98e5d6a88ab4a26b4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34721: connect: Connection refused (error 111)
I20260924 04:36:00.222373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 2474
I20260924 04:36:00.228976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 2090
2026-09-24T04:36:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6247 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:00 dist-test-slave-jlnt krb5kdc[2778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:00 dist-test-slave-jlnt krb5kdc[2778](info): set up 2 sockets
Sep 24 04:36:00 dist-test-slave-jlnt krb5kdc[2778](info): commencing operation
krb5kdc: starting...
W20260924 04:36:01.281299 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:01 dist-test-slave-jlnt krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:36:01.400892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:01.508525 2791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:01.508801 2791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:01.508881 2791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:01.512840 2791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:01.512923 2791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:01.512959 2791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:01.513015 2791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:01.513051 2791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:01.518188 2791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:01.519155 2791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:01.520124 2791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:01.528175 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:01.528167 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:01.528256 2791 server_base.cc:1034] running on GCE node
W20260924 04:36:01.528214 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:01.529006 2791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:01.530032 2791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:01.531185 2791 hybrid_clock.cc:648] HybridClock initialized: now 1790224561531156 us; error 42 us; skew 500 ppm
Sep 24 04:36:01 dist-test-slave-jlnt krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:01.533937 2791 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:36:01.534778 2791 webserver.cc:460] Webserver started at http://127.0.98.62:36937/ using document root <none> and password file <none>
I20260924 04:36:01.535308 2791 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:01.535395 2791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:01.535633 2791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:01.537611 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "3aa9a249d1bb4370b84e6c0006dc6581"
format_stamp: "Formatted at 2026-09-24 04:36:01 on dist-test-slave-jlnt"
server_key: "bb3712aedd231895a8acfc2d9be45907"
server_key_iv: "7a4ffe2b14a770280943d5af7e4a48c6"
server_key_version: "encryptionkey@0"
I20260924 04:36:01.538156 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "3aa9a249d1bb4370b84e6c0006dc6581"
format_stamp: "Formatted at 2026-09-24 04:36:01 on dist-test-slave-jlnt"
server_key: "bb3712aedd231895a8acfc2d9be45907"
server_key_iv: "7a4ffe2b14a770280943d5af7e4a48c6"
server_key_version: "encryptionkey@0"
I20260924 04:36:01.541641 2791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:01.543957 2806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:01.545394 2791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:01.545534 2791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "3aa9a249d1bb4370b84e6c0006dc6581"
format_stamp: "Formatted at 2026-09-24 04:36:01 on dist-test-slave-jlnt"
server_key: "bb3712aedd231895a8acfc2d9be45907"
server_key_iv: "7a4ffe2b14a770280943d5af7e4a48c6"
server_key_version: "encryptionkey@0"
I20260924 04:36:01.545662 2791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:01.569025 2791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:01.572464 2791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:01.572677 2791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:01.579806 2791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40707
I20260924 04:36:01.579823 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40707 every 8 connection(s)
I20260924 04:36:01.580737 2791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:01.583385 2859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:01.586427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 2791
I20260924 04:36:01.586544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:01.586812 392 external_mini_cluster.cc:1428] Setting key 911d3884f70932bf8286d607b1ce732d
I20260924 04:36:01.589094 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581: Bootstrap starting.
I20260924 04:36:01.591766 2859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:01.592593 2859 log.cc:824] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:01.594367 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581: No bootstrap required, opened a new log
Sep 24 04:36:01 dist-test-slave-jlnt krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:01.598106 2859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9a249d1bb4370b84e6c0006dc6581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40707 } }
I20260924 04:36:01.598307 2859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:01.598340 2859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa9a249d1bb4370b84e6c0006dc6581, State: Initialized, Role: FOLLOWER
I20260924 04:36:01.598810 2859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9a249d1bb4370b84e6c0006dc6581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40707 } }
I20260924 04:36:01.598925 2859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:01.599026 2859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:01.599140 2859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:01.599948 2859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9a249d1bb4370b84e6c0006dc6581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40707 } }
I20260924 04:36:01.600289 2859 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aa9a249d1bb4370b84e6c0006dc6581; no voters:
I20260924 04:36:01.600631 2859 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:01.600689 2864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:01.600883 2864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [term 1 LEADER]: Becoming Leader. State: Replica: 3aa9a249d1bb4370b84e6c0006dc6581, State: Running, Role: LEADER
I20260924 04:36:01.601207 2864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9a249d1bb4370b84e6c0006dc6581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40707 } }
I20260924 04:36:01.601481 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:01.588184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.0.98.62:40707)
0924 04:36:01.588583 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:01.588601 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:01.588641 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:36:01.589479 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:01.589503 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:01.589898 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:01.590258 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:01.590911 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:01.592391 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:01.593339 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:01.593817 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:01.593988 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:01.596598 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:01.596603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:01.596615 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:01.596640 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:01.598438 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:01.598866 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:01.598871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:01.598877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:01.599074 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:01.599353 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:01.599357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:01.599359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:01.599730 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:01.599938 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:01.600166 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 04:36:01.600489 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":107,"threads_started":1}
I20260924 04:36:01.601863 2859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:01.602764 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aa9a249d1bb4370b84e6c0006dc6581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9a249d1bb4370b84e6c0006dc6581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40707 } } }
I20260924 04:36:01.602905 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:01.603693 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa9a249d1bb4370b84e6c0006dc6581. Latest consensus state: current_term: 1 leader_uuid: "3aa9a249d1bb4370b84e6c0006dc6581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa9a249d1bb4370b84e6c0006dc6581" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40707 } } }
I20260924 04:36:01.603802 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:01.603894 2873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:01.606796 2873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:01.611760 2873 catalog_manager.cc:1347] Generated new cluster ID: 57ec2c0bcffa4492b49c6abe8d7f96a2
I20260924 04:36:01.611821 2873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:01.619634 2873 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:01.620785 2873 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:01.625800 2873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aa9a249d1bb4370b84e6c0006dc6581: Generated new TSK 0
I20260924 04:36:01.626508 2873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:36:01.654170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40707
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:01.765450 2885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:01.765690 2885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:01.765731 2885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:01.769913 2885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:01.769985 2885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:01.770095 2885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:01.775595 2885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:01.776603 2885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:01.777673 2885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:01.786945 2891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:01.786909 2890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:01.786927 2893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:01.786975 2885 server_base.cc:1034] running on GCE node
I20260924 04:36:01.787563 2885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:01.788096 2885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:01.789211 2885 hybrid_clock.cc:648] HybridClock initialized: now 1790224561789198 us; error 25 us; skew 500 ppm
Sep 24 04:36:01 dist-test-slave-jlnt krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:01.791781 2885 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:36:01.792557 2885 webserver.cc:460] Webserver started at http://127.0.98.1:35577/ using document root <none> and password file <none>
I20260924 04:36:01.793080 2885 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:01.793195 2885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:01.793422 2885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:01.795372 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "93e5c27f6b28457da45899e79851d5b9"
format_stamp: "Formatted at 2026-09-24 04:36:01 on dist-test-slave-jlnt"
server_key: "25b0af897a575c326e87dd4b9a5287de"
server_key_iv: "5a90dfa55d342d138eefeb3c7628c05b"
server_key_version: "encryptionkey@0"
I20260924 04:36:01.795917 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "93e5c27f6b28457da45899e79851d5b9"
format_stamp: "Formatted at 2026-09-24 04:36:01 on dist-test-slave-jlnt"
server_key: "25b0af897a575c326e87dd4b9a5287de"
server_key_iv: "5a90dfa55d342d138eefeb3c7628c05b"
server_key_version: "encryptionkey@0"
I20260924 04:36:01.799371 2885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:36:01.801713 2900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:01.803011 2885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:01.803128 2885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "93e5c27f6b28457da45899e79851d5b9"
format_stamp: "Formatted at 2026-09-24 04:36:01 on dist-test-slave-jlnt"
server_key: "25b0af897a575c326e87dd4b9a5287de"
server_key_iv: "5a90dfa55d342d138eefeb3c7628c05b"
server_key_version: "encryptionkey@0"
I20260924 04:36:01.803236 2885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:01.834116 2885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:01.836974 2885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:01.837188 2885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:01.837785 2885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:01.838567 2885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:01.838637 2885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:01.838704 2885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:01.838757 2885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:01.848104 2885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36123
I20260924 04:36:01.848127 3013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36123 every 8 connection(s)
I20260924 04:36:01.849030 2885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:01.850260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 2885
I20260924 04:36:01.850373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:01.850661 392 external_mini_cluster.cc:1428] Setting key 0f9a85a3507d761844adf761b078adf4
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 24 04:36:01 dist-test-slave-jlnt krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:36:01.864914 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:01.851055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38389 (local address 127.0.98.62:40707)
0924 04:36:01.851189 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:01.851196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:01.852203 (+ 1007us) server_negotiation.cc:407] Connection header received
0924 04:36:01.853679 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:01.853683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:01.853738 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:01.853806 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:01.855383 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:01.856182 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:01.858064 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:01.858285 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:01.858364 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:01.861188 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:01.861191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:01.861194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:01.861220 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:01.862804 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:01.863392 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:01.863397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:01.863399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:01.863483 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:01.863815 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:01.863819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:01.863821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:01.864008 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:01.864117 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:01.864618 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 04:36:01.864738 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:36:01.865739 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:01.851518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40707 (local address 127.0.98.1:38389)
0924 04:36:01.852054 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:01.852111 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:01.853426 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:01.854029 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:01.854042 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:01.854538 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:01.855229 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:01.855252 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:01.856320 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:01.856327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:01.857927 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:01.857938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:01.858435 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:01.858439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:01.858648 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:01.859256 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:01.859279 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:01.860997 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:01.862947 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:01.862953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:01.862967 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:01.863288 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:01.863575 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:01.863580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:01.863582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:01.863721 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:01.864122 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:01.864131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:01.864376 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 04:36:01.864565 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 04:36:01.864880 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":123,"threads_started":1}
I20260924 04:36:01.867131 3014 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40707
I20260924 04:36:01.867398 3014 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:01.867892 3014 heartbeater.cc:511] Master 127.0.98.62:40707 requested a full tablet report, sending...
I20260924 04:36:01.869585 2823 ts_manager.cc:194] Registered new tserver with Master: 93e5c27f6b28457da45899e79851d5b9 (127.0.98.1:36123)
I20260924 04:36:01.871069 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:36:01.880048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40707
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:01.985353 3019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:01.985582 3019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:01.985623 3019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:01.989511 3019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:01.989562 3019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:01.989616 3019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:01.994781 3019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:01.995712 3019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:01.996657 3019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:02.005961 3025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:02.005957 3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:02.006068 3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:02.006311 3019 server_base.cc:1034] running on GCE node
I20260924 04:36:02.006757 3019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:02.007424 3019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:02.008590 3019 hybrid_clock.cc:648] HybridClock initialized: now 1790224562008570 us; error 42 us; skew 500 ppm
Sep 24 04:36:02 dist-test-slave-jlnt krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:02.012053 3019 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:36:02.013224 3019 webserver.cc:460] Webserver started at http://127.0.98.2:43315/ using document root <none> and password file <none>
I20260924 04:36:02.013967 3019 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:02.014055 3019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:02.014304 3019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:02.017006 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "d995a22f0d784a66aa347daa4afc62e4"
format_stamp: "Formatted at 2026-09-24 04:36:02 on dist-test-slave-jlnt"
server_key: "ffb0d83e2f511f0b940591a85d3ac562"
server_key_iv: "68f616c3204f6898b59468dbca39d974"
server_key_version: "encryptionkey@0"
I20260924 04:36:02.017719 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d995a22f0d784a66aa347daa4afc62e4"
format_stamp: "Formatted at 2026-09-24 04:36:02 on dist-test-slave-jlnt"
server_key: "ffb0d83e2f511f0b940591a85d3ac562"
server_key_iv: "68f616c3204f6898b59468dbca39d974"
server_key_version: "encryptionkey@0"
I20260924 04:36:02.021445 3019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:02.023823 3034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:02.025010 3019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:02.025172 3019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "d995a22f0d784a66aa347daa4afc62e4"
format_stamp: "Formatted at 2026-09-24 04:36:02 on dist-test-slave-jlnt"
server_key: "ffb0d83e2f511f0b940591a85d3ac562"
server_key_iv: "68f616c3204f6898b59468dbca39d974"
server_key_version: "encryptionkey@0"
I20260924 04:36:02.025288 3019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:02.047178 3019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:02.050258 3019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:02.050446 3019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:02.051012 3019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:02.051860 3019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:02.051925 3019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:02.051990 3019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:02.052035 3019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:02.060775 3019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40447
I20260924 04:36:02.060815 3147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40447 every 8 connection(s)
I20260924 04:36:02.061721 3019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:02.065852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 3019
I20260924 04:36:02.065980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:02.066329 392 external_mini_cluster.cc:1428] Setting key d59af214057b3521be2fbb827710ef48
Sep 24 04:36:02 dist-test-slave-jlnt krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:36:02.077333 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:02.063436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48361 (local address 127.0.98.62:40707)
0924 04:36:02.063611 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:02.063615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:02.064410 (+ 795us) server_negotiation.cc:407] Connection header received
0924 04:36:02.066032 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:02.066037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:02.066110 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:02.066195 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:02.067633 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:02.068364 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:02.070007 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:02.070389 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:02.070519 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:02.073401 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:02.073407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:02.073413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:02.073449 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:02.075214 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:02.075740 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:02.075747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:02.075752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:02.075845 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:02.076176 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:02.076182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:02.076185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:02.076435 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:02.076556 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:02.077016 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 04:36:02.077118 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:36:02.078326 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:02.063818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40707 (local address 127.0.98.2:48361)
0924 04:36:02.064253 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:02.064302 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:02.065734 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:02.066410 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:02.066420 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:02.066910 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:02.067441 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:02.067479 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:02.068541 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:02.068547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:02.069881 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:02.069890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:02.070566 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:02.070569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:02.070709 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:02.071344 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:02.071360 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:02.073234 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:02.075329 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:02.075334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:02.075347 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:02.075614 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:02.075939 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:02.075941 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:02.075942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:02.076076 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:02.076550 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:02.076556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:02.076790 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 04:36:02.076945 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:36:02.077347 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:36:02.079504 3148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40707
I20260924 04:36:02.079766 3148 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:02.080333 3148 heartbeater.cc:511] Master 127.0.98.62:40707 requested a full tablet report, sending...
I20260924 04:36:02.081518 2823 ts_manager.cc:194] Registered new tserver with Master: d995a22f0d784a66aa347daa4afc62e4 (127.0.98.2:40447)
I20260924 04:36:02.082475 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:36:02.097251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40707
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:02.205965 3153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:02.206203 3153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:02.206244 3153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:02.210424 3153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:02.210485 3153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:02.210542 3153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:02.215667 3153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:02.216595 3153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:02.217547 3153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:02.226732 3159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:02.226689 3158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:02.227015 3161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:02.226975 3153 server_base.cc:1034] running on GCE node
I20260924 04:36:02.227451 3153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:02.228094 3153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:02.229218 3153 hybrid_clock.cc:648] HybridClock initialized: now 1790224562229198 us; error 31 us; skew 500 ppm
Sep 24 04:36:02 dist-test-slave-jlnt krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:02.231861 3153 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:36:02.232688 3153 webserver.cc:460] Webserver started at http://127.0.98.3:43649/ using document root <none> and password file <none>
I20260924 04:36:02.233255 3153 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:02.233319 3153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:02.233495 3153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:02.235379 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "f82275bebf314ac6baf602d21ff73126"
format_stamp: "Formatted at 2026-09-24 04:36:02 on dist-test-slave-jlnt"
server_key: "72ba65b2eb67cefe513fd50b196f43ae"
server_key_iv: "fa3d8f8c8ef4cee4857ae67f689596b3"
server_key_version: "encryptionkey@0"
I20260924 04:36:02.235859 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f82275bebf314ac6baf602d21ff73126"
format_stamp: "Formatted at 2026-09-24 04:36:02 on dist-test-slave-jlnt"
server_key: "72ba65b2eb67cefe513fd50b196f43ae"
server_key_iv: "fa3d8f8c8ef4cee4857ae67f689596b3"
server_key_version: "encryptionkey@0"
I20260924 04:36:02.239360 3153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:02.241607 3168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:02.242772 3153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:02.242883 3153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "f82275bebf314ac6baf602d21ff73126"
format_stamp: "Formatted at 2026-09-24 04:36:02 on dist-test-slave-jlnt"
server_key: "72ba65b2eb67cefe513fd50b196f43ae"
server_key_iv: "fa3d8f8c8ef4cee4857ae67f689596b3"
server_key_version: "encryptionkey@0"
I20260924 04:36:02.242981 3153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:02.271932 3153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:02.274890 3153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:02.275091 3153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:02.275673 3153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:02.276469 3153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:02.276513 3153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:02.276585 3153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:02.276623 3153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:02.285248 3153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37405
I20260924 04:36:02.285274 3281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37405 every 8 connection(s)
I20260924 04:36:02.286091 3153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:02.293201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 3153
I20260924 04:36:02.293323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:02.293607 392 external_mini_cluster.cc:1428] Setting key 58904f98c14de4d47b15ff2133456984
Sep 24 04:36:02 dist-test-slave-jlnt krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:02.298956 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:02.287728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41565 (local address 127.0.98.62:40707)
0924 04:36:02.287863 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:02.287873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:02.288578 (+ 705us) server_negotiation.cc:407] Connection header received
0924 04:36:02.289954 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:02.289957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:02.290016 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:02.290092 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:02.291193 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:02.291721 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:02.292839 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:02.293080 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:02.293218 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:02.295997 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:02.296000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:02.296002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:02.296024 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:02.297278 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:02.297715 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:02.297717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:02.297719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:02.297782 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:02.298054 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:02.298056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:02.298057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:02.298218 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:02.298290 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:02.298767 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 04:36:02.298848 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:36:02.299757 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:02.288024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40707 (local address 127.0.98.3:41565)
0924 04:36:02.288430 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:02.288469 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:02.289749 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:02.290252 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:02.290259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:02.290596 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:02.291044 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:02.291057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:02.291858 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:02.291861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:02.292708 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:02.292716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:02.293249 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:02.293253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:02.293475 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:02.294063 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:02.294087 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:02.295861 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:02.297374 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:02.297378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:02.297388 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:02.297634 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:02.297856 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:02.297857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:02.297859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:02.297980 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:02.298332 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:02.298337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:02.298564 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 04:36:02.298728 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:36:02.298954 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260924 04:36:02.300806 3282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40707
I20260924 04:36:02.300999 3282 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:02.301450 3282 heartbeater.cc:511] Master 127.0.98.62:40707 requested a full tablet report, sending...
I20260924 04:36:02.302407 2823 ts_manager.cc:194] Registered new tserver with Master: f82275bebf314ac6baf602d21ff73126 (127.0.98.3:37405)
I20260924 04:36:02.303023 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41565
I20260924 04:36:02.307269 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:36:02.319060 2862 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:02.317920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.0.98.62:40707)
0924 04:36:02.318043 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:02.318046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:02.318134 (+ 88us) server_negotiation.cc:407] Connection header received
0924 04:36:02.318229 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:02.318231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:02.318269 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:02.318338 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:02.318954 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50566: BlockingRecv error: recv got EOF from 127.0.0.1:50566 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:36:02.320039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 2885
I20260924 04:36:02.325531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 3019
I20260924 04:36:02.330636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 3153
I20260924 04:36:02.335799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 2791
2026-09-24T04:36:02Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2105 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:02 dist-test-slave-jlnt krb5kdc[3294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:02 dist-test-slave-jlnt krb5kdc[3294](info): set up 2 sockets
Sep 24 04:36:02 dist-test-slave-jlnt krb5kdc[3294](info): commencing operation
krb5kdc: starting...
W20260924 04:36:03.369105 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:03 dist-test-slave-jlnt krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:36:03.488243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33625
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:03.601414 3307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:03.601668 3307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:03.601708 3307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:03.605623 3307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:03.605676 3307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:03.605695 3307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:03.605712 3307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:03.605727 3307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:03.610819 3307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:03.611729 3307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:03.612639 3307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:03.621060 3313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:03.621042 3312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:03.621061 3315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:03.621343 3307 server_base.cc:1034] running on GCE node
I20260924 04:36:03.621892 3307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:03.622903 3307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:03.624106 3307 hybrid_clock.cc:648] HybridClock initialized: now 1790224563624094 us; error 34 us; skew 500 ppm
Sep 24 04:36:03 dist-test-slave-jlnt krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:03.626685 3307 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:36:03.627518 3307 webserver.cc:460] Webserver started at http://127.0.98.62:41217/ using document root <none> and password file <none>
I20260924 04:36:03.628042 3307 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:03.628130 3307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:03.628353 3307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:03.630332 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "d387037777ae4996ae50e939d9c10089"
format_stamp: "Formatted at 2026-09-24 04:36:03 on dist-test-slave-jlnt"
server_key: "aa65a6895faf1a43914ec272f482e0ec"
server_key_iv: "f38a6223294f582ef5e0a83728a60b9c"
server_key_version: "encryptionkey@0"
I20260924 04:36:03.630864 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "d387037777ae4996ae50e939d9c10089"
format_stamp: "Formatted at 2026-09-24 04:36:03 on dist-test-slave-jlnt"
server_key: "aa65a6895faf1a43914ec272f482e0ec"
server_key_iv: "f38a6223294f582ef5e0a83728a60b9c"
server_key_version: "encryptionkey@0"
I20260924 04:36:03.634282 3307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:03.636443 3322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:03.637658 3307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:03.637804 3307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "d387037777ae4996ae50e939d9c10089"
format_stamp: "Formatted at 2026-09-24 04:36:03 on dist-test-slave-jlnt"
server_key: "aa65a6895faf1a43914ec272f482e0ec"
server_key_iv: "f38a6223294f582ef5e0a83728a60b9c"
server_key_version: "encryptionkey@0"
I20260924 04:36:03.637905 3307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:03.655748 3307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:03.658568 3307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:03.658773 3307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:03.665568 3307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33625
I20260924 04:36:03.665845 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33625 every 8 connection(s)
I20260924 04:36:03.666484 3307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:03.669116 3375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:03.673707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 3307
I20260924 04:36:03.673836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:03.674110 392 external_mini_cluster.cc:1428] Setting key 804f8ca375853069bb64e858dea8cac6
I20260924 04:36:03.674717 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089: Bootstrap starting.
I20260924 04:36:03.676997 3375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:03.677910 3375 log.cc:824] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:03.679883 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089: No bootstrap required, opened a new log
Sep 24 04:36:03 dist-test-slave-jlnt krb5kdc[3294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:03.684376 3375 raft_consensus.cc:348] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d387037777ae4996ae50e939d9c10089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } }
I20260924 04:36:03.684625 3375 raft_consensus.cc:374] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:03.684716 3375 raft_consensus.cc:729] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d387037777ae4996ae50e939d9c10089, State: Initialized, Role: FOLLOWER
I20260924 04:36:03.685174 3375 consensus_queue.cc:260] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d387037777ae4996ae50e939d9c10089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } }
I20260924 04:36:03.685309 3375 raft_consensus.cc:388] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:03.685387 3375 raft_consensus.cc:482] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:03.685475 3375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:03.686311 3375 raft_consensus.cc:504] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d387037777ae4996ae50e939d9c10089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } }
I20260924 04:36:03.686647 3375 leader_election.cc:302] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d387037777ae4996ae50e939d9c10089; no voters:
I20260924 04:36:03.686972 3375 leader_election.cc:288] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:03.687037 3380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:03.687259 3380 raft_consensus.cc:686] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [term 1 LEADER]: Becoming Leader. State: Replica: d387037777ae4996ae50e939d9c10089, State: Running, Role: LEADER
I20260924 04:36:03.687628 3380 consensus_queue.cc:237] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d387037777ae4996ae50e939d9c10089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } }
I20260924 04:36:03.688284 3375 sys_catalog.cc:564] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:03.688670 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:03.675329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.0.98.62:33625)
0924 04:36:03.675710 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:03.675725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:03.675759 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:36:03.676334 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:03.676349 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:03.676646 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:03.676920 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:03.677844 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:03.678812 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:03.679925 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:03.680408 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:03.680526 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:03.683465 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:03.683470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:03.683482 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:03.683506 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:03.685478 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:03.685945 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:03.685950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:03.685956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:03.686073 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:03.686350 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:03.686354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:03.686356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:03.686802 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:03.687016 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:03.687603 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 04:36:03.687880 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":120,"threads_started":1}
I20260924 04:36:03.689415 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d387037777ae4996ae50e939d9c10089. Latest consensus state: current_term: 1 leader_uuid: "d387037777ae4996ae50e939d9c10089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d387037777ae4996ae50e939d9c10089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } } }
I20260924 04:36:03.689541 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:03.689462 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d387037777ae4996ae50e939d9c10089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d387037777ae4996ae50e939d9c10089" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33625 } } }
I20260924 04:36:03.689589 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:03.689888 3389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:03.691995 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:36:03.693783 3389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:03.701095 3389 catalog_manager.cc:1347] Generated new cluster ID: 743a0e866506419d8fbe620e089531b6
I20260924 04:36:03.701222 3389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:03.708278 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:03.703714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.0.98.62:33625)
0924 04:36:03.703834 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:03.703838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:03.704072 (+ 234us) server_negotiation.cc:407] Connection header received
0924 04:36:03.704239 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:03.704242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:03.704290 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:03.704377 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:03.704862 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:03.705427 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:03.706249 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:03.706490 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:03.706608 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:03.707455 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:03.707458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:03.707461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:03.707857 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:03.707883 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:03.707899 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:03.707992 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:03.708086 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:36:03.708141 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:36:03.710500 3389 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:03.711323 3389 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:03.720530 3389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d387037777ae4996ae50e939d9c10089: Generated new TSK 0
I20260924 04:36:03.721258 3389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:36:03.730388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 3307
2026-09-24T04:36:03Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1392 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:03 dist-test-slave-jlnt krb5kdc[3408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:03 dist-test-slave-jlnt krb5kdc[3408](info): set up 2 sockets
Sep 24 04:36:03 dist-test-slave-jlnt krb5kdc[3408](info): commencing operation
krb5kdc: starting...
W20260924 04:36:04.761122 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:04 dist-test-slave-jlnt krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:36:04.880810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44569
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:04.986086 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:04.986346 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:04.986387 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:04.990347 3421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:04.990401 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:04.990420 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:04.990437 3421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:04.990451 3421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:04.995496 3421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:04.996483 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:04.997481 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:05.005483 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:05.005610 3421 server_base.cc:1034] running on GCE node
W20260924 04:36:05.005506 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:05.005826 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:05.006397 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:05.007455 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:05.008641 3421 hybrid_clock.cc:648] HybridClock initialized: now 1790224565008621 us; error 33 us; skew 500 ppm
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:05.011382 3421 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:36:05.012252 3421 webserver.cc:460] Webserver started at http://127.0.98.62:45025/ using document root <none> and password file <none>
I20260924 04:36:05.012809 3421 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:05.012899 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:05.013119 3421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:05.015067 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "95d093769d794055a2296de3ffca16d1"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "a9f1d5fa254418f4fc9d73a4190f4b34"
server_key_iv: "6e2d6dc628548111ae88cb0ed1ff6ac4"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.015621 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "95d093769d794055a2296de3ffca16d1"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "a9f1d5fa254418f4fc9d73a4190f4b34"
server_key_iv: "6e2d6dc628548111ae88cb0ed1ff6ac4"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.019239 3421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:05.021613 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.022838 3421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:05.022971 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "95d093769d794055a2296de3ffca16d1"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "a9f1d5fa254418f4fc9d73a4190f4b34"
server_key_iv: "6e2d6dc628548111ae88cb0ed1ff6ac4"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.023077 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:05.045383 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:05.048656 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:05.048872 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:05.055596 3421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44569
I20260924 04:36:05.055965 3488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44569 every 8 connection(s)
I20260924 04:36:05.056559 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:05.059271 3489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:05.064966 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1: Bootstrap starting.
I20260924 04:36:05.066327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 3421
I20260924 04:36:05.066448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:05.066744 392 external_mini_cluster.cc:1428] Setting key 83dbffd00f6e32ded6b7598e3325611e
I20260924 04:36:05.067919 3489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:05.068717 3489 log.cc:824] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:05.070737 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1: No bootstrap required, opened a new log
I20260924 04:36:05.074442 3489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d093769d794055a2296de3ffca16d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44569 } }
I20260924 04:36:05.074707 3489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:05.074791 3489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d093769d794055a2296de3ffca16d1, State: Initialized, Role: FOLLOWER
I20260924 04:36:05.075294 3489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d093769d794055a2296de3ffca16d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44569 } }
I20260924 04:36:05.075477 3489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:05.075551 3489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:05.075649 3489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:05.076906 3489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d093769d794055a2296de3ffca16d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44569 } }
I20260924 04:36:05.077345 3489 leader_election.cc:302] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95d093769d794055a2296de3ffca16d1; no voters:
I20260924 04:36:05.077720 3489 leader_election.cc:288] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:05.077839 3494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:05.078068 3494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [term 1 LEADER]: Becoming Leader. State: Replica: 95d093769d794055a2296de3ffca16d1, State: Running, Role: LEADER
I20260924 04:36:05.078437 3494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d093769d794055a2296de3ffca16d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44569 } }
I20260924 04:36:05.079095 3489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:05.079994 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d093769d794055a2296de3ffca16d1. Latest consensus state: current_term: 1 leader_uuid: "95d093769d794055a2296de3ffca16d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d093769d794055a2296de3ffca16d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44569 } } }
I20260924 04:36:05.080160 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:05.080482 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95d093769d794055a2296de3ffca16d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d093769d794055a2296de3ffca16d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44569 } } }
I20260924 04:36:05.080588 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:05.080929 3503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:05.082599 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:05.067896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.0.98.62:44569)
0924 04:36:05.068352 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:05.068368 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:05.068404 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:36:05.069226 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:05.069250 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:05.069678 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:05.070023 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:05.070845 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.071866 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.072730 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.073309 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.073480 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:05.076698 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:05.076704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:05.076718 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:05.076748 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:05.078863 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.079428 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.079436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.079442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.079598 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.079902 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.079907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.079908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.080475 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:05.080706 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:05.081327 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 04:36:05.081626 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":152,"threads_started":1}
I20260924 04:36:05.084169 3503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:05.089982 3503 catalog_manager.cc:1347] Generated new cluster ID: 905cc725358e44749ebbc0e2e88c2bbc
I20260924 04:36:05.090058 3503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:05.125737 3503 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:05.126771 3503 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:05.135829 3503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95d093769d794055a2296de3ffca16d1: Generated new TSK 0
I20260924 04:36:05.136848 3503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:36:05.180966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44569
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:05.300874 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:05.301115 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:05.301224 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:05.305042 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:05.305088 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:05.305160 3515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:05.310213 3515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:05.311105 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:05.312031 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:05.320976 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:05.321029 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:05.321206 3515 server_base.cc:1034] running on GCE node
W20260924 04:36:05.321065 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:05.321688 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:05.322268 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:05.323426 3515 hybrid_clock.cc:648] HybridClock initialized: now 1790224565323408 us; error 28 us; skew 500 ppm
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:05.338238 3515 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:36:05.339250 3515 webserver.cc:460] Webserver started at http://127.0.98.1:37867/ using document root <none> and password file <none>
I20260924 04:36:05.339877 3515 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:05.339977 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:05.340230 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:05.342520 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "075068cafc9f4129a56ef7b320eb01d3"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "df8cbe740dc30a54f4b35ecf723d64f8"
server_key_iv: "8aab15c185d62a1d8ee49b1436f07a23"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.343158 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "075068cafc9f4129a56ef7b320eb01d3"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "df8cbe740dc30a54f4b35ecf723d64f8"
server_key_iv: "8aab15c185d62a1d8ee49b1436f07a23"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.347199 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:36:05.349769 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.351080 3515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:05.351212 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "075068cafc9f4129a56ef7b320eb01d3"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "df8cbe740dc30a54f4b35ecf723d64f8"
server_key_iv: "8aab15c185d62a1d8ee49b1436f07a23"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.351325 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:05.366943 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:05.370167 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:05.370370 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:05.370961 3515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:05.371798 3515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:05.371867 3515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.371939 3515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:05.371982 3515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.381345 3515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34821
I20260924 04:36:05.381379 3643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34821 every 8 connection(s)
I20260924 04:36:05.382283 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:05.387871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 3515
I20260924 04:36:05.388000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:05.388316 392 external_mini_cluster.cc:1428] Setting key f5a6945e27e9207ede9974e558174ed2
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:05.398074 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:05.384145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57061 (local address 127.0.98.62:44569)
0924 04:36:05.384294 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:05.384299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:05.385070 (+ 771us) server_negotiation.cc:407] Connection header received
0924 04:36:05.386800 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:05.386804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:05.386863 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:05.386942 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:05.388386 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.388945 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.390496 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.390739 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.390833 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:05.393974 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:05.393977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:05.393979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:05.394001 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:05.395803 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.396310 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.396315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.396319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.396418 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.396746 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.396752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.396755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.397014 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:05.397117 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:05.397711 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 04:36:05.397850 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:36:05.398772 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:05.384424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44569 (local address 127.0.98.1:57061)
0924 04:36:05.384898 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:05.384940 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:05.386461 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:05.387118 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:05.387126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:05.387482 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:05.388188 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:05.388210 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.389070 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.389073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:05.390362 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:05.390376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.390899 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.390906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:05.391149 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:05.391901 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:05.391917 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:05.393774 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:05.395924 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:05.395928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:05.395940 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:05.396194 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:05.396509 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:05.396511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:05.396512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:05.396647 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:05.397124 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:05.397129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:05.397415 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 04:36:05.397597 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:36:05.397845 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
I20260924 04:36:05.400049 3644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44569
I20260924 04:36:05.400290 3644 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:05.400818 3644 heartbeater.cc:511] Master 127.0.98.62:44569 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:36:05.402458 3453 ts_manager.cc:194] Registered new tserver with Master: 075068cafc9f4129a56ef7b320eb01d3 (127.0.98.1:34821)
I20260924 04:36:05.403987 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:36:05.417272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44569
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:05.524590 3649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:05.524832 3649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:05.524873 3649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:05.528940 3649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:05.528993 3649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:05.529049 3649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:05.534176 3649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:05.535127 3649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:05.536062 3649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:05.545295 3657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:05.545320 3655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:05.545337 3654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:05.545513 3649 server_base.cc:1034] running on GCE node
I20260924 04:36:05.546007 3649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:05.546594 3649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:05.547744 3649 hybrid_clock.cc:648] HybridClock initialized: now 1790224565547712 us; error 45 us; skew 500 ppm
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:05.550426 3649 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:36:05.551373 3649 webserver.cc:460] Webserver started at http://127.0.98.2:33489/ using document root <none> and password file <none>
I20260924 04:36:05.551909 3649 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:05.551968 3649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:05.552175 3649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:05.554147 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "198528d9e28a4682b8737695a1980b81"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "ba4736385bec27d9e96aa1d3289d993a"
server_key_iv: "e2d583d863610dc6b52c8fd8fc76c431"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.554639 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "198528d9e28a4682b8737695a1980b81"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "ba4736385bec27d9e96aa1d3289d993a"
server_key_iv: "e2d583d863610dc6b52c8fd8fc76c431"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.558128 3649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:05.560298 3664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.561540 3649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:05.561657 3649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "198528d9e28a4682b8737695a1980b81"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "ba4736385bec27d9e96aa1d3289d993a"
server_key_iv: "e2d583d863610dc6b52c8fd8fc76c431"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.561767 3649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:05.583400 3649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:05.586745 3649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:05.586932 3649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:05.587482 3649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:05.588253 3649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:05.588299 3649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.588337 3649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:05.588389 3649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.596657 3649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38711
I20260924 04:36:05.596693 3777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38711 every 8 connection(s)
I20260924 04:36:05.597582 3649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:05.602802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 3649
I20260924 04:36:05.602927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:05.603201 392 external_mini_cluster.cc:1428] Setting key 906d1c1271c60df3c3408bf902b7b310
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:05.611268 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:05.599266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52307 (local address 127.0.98.62:44569)
0924 04:36:05.599400 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:05.599404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:05.600227 (+ 823us) server_negotiation.cc:407] Connection header received
0924 04:36:05.601629 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:05.601632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:05.601677 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:05.601749 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:05.602894 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.603626 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.604987 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.605412 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.605531 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:05.607898 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:05.607907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:05.607910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:05.607930 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:05.609367 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.609847 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.609851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.609853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.609928 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.610246 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.610251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.610253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.610417 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:05.610513 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:05.610983 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 04:36:05.611103 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:36:05.612102 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:05.599682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44569 (local address 127.0.98.2:52307)
0924 04:36:05.600092 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:05.600130 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:05.601415 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:05.601898 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:05.601904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:05.602241 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:05.602762 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:05.602774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.603756 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.603759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:05.604875 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:05.604884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.605572 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.605575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:05.605721 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:05.606352 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:05.606367 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:05.607747 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:05.609483 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:05.609488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:05.609502 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:05.609751 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:05.610014 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:05.610018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:05.610021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:05.610158 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:05.610520 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:05.610527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:05.610760 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 04:36:05.610931 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 04:36:05.611231 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":108,"threads_started":1}
I20260924 04:36:05.613252 3778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44569
I20260924 04:36:05.613476 3778 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:05.613976 3778 heartbeater.cc:511] Master 127.0.98.62:44569 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:36:05.615003 3453 ts_manager.cc:194] Registered new tserver with Master: 198528d9e28a4682b8737695a1980b81 (127.0.98.2:38711)
I20260924 04:36:05.615826 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:36:05.630211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44569
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:05.735329 3783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:05.735611 3783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:05.735688 3783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:05.739614 3783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:05.739699 3783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:05.739804 3783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:05.744863 3783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:05.745924 3783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:05.746935 3783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:05.756270 3788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:05.756321 3791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:05.756323 3783 server_base.cc:1034] running on GCE node
W20260924 04:36:05.756274 3789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:05.756860 3783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:05.757483 3783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:05.758656 3783 hybrid_clock.cc:648] HybridClock initialized: now 1790224565758639 us; error 28 us; skew 500 ppm
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:05.761233 3783 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:36:05.762049 3783 webserver.cc:460] Webserver started at http://127.0.98.3:33961/ using document root <none> and password file <none>
I20260924 04:36:05.762571 3783 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:05.762655 3783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:05.762877 3783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:05.764801 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "74b40d0ce3b942dbba0b20f9e85bcc8b"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "5f897cb89c61d6b1737080db1e1a0e14"
server_key_iv: "3a22d83ec829f07959f14dccaf5d89d9"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.765385 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "74b40d0ce3b942dbba0b20f9e85bcc8b"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "5f897cb89c61d6b1737080db1e1a0e14"
server_key_iv: "3a22d83ec829f07959f14dccaf5d89d9"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.768863 3783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:36:05.771227 3798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.772444 3783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:05.772614 3783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "74b40d0ce3b942dbba0b20f9e85bcc8b"
format_stamp: "Formatted at 2026-09-24 04:36:05 on dist-test-slave-jlnt"
server_key: "5f897cb89c61d6b1737080db1e1a0e14"
server_key_iv: "3a22d83ec829f07959f14dccaf5d89d9"
server_key_version: "encryptionkey@0"
I20260924 04:36:05.772722 3783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:05.786818 3783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:05.789538 3783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:05.789731 3783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:05.790283 3783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:05.791061 3783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:05.791126 3783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.791194 3783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:05.791234 3783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:05.800170 3783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33049
I20260924 04:36:05.800196 3911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33049 every 8 connection(s)
I20260924 04:36:05.801018 3783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:05.805536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 3783
I20260924 04:36:05.805635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:05.805919 392 external_mini_cluster.cc:1428] Setting key 75a35692b64bfc9b595aaaf13430243e
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:05.814369 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:05.802844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45717 (local address 127.0.98.62:44569)
0924 04:36:05.802969 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:05.802972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:05.803793 (+ 821us) server_negotiation.cc:407] Connection header received
0924 04:36:05.805116 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:05.805120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:05.805192 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:05.805273 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:05.806665 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.807460 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.808768 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.808974 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.809057 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:05.811415 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:05.811418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:05.811420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:05.811442 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:05.812652 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.813128 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.813130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.813132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.813225 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.813504 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.813506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.813507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.813661 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:05.813750 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:05.814168 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 04:36:05.814256 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:36:05.815136 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:05.803209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44569 (local address 127.0.98.3:45717)
0924 04:36:05.803641 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:05.803682 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:05.804899 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:05.805448 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:05.805455 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:05.805893 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:05.806496 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:05.806512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.807614 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.807617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:05.808662 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:05.808671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.809099 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.809101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:05.809298 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:05.809863 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:05.809878 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:05.811267 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:05.812780 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:05.812784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:05.812794 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:05.813048 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:05.813299 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:05.813301 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:05.813302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:05.813428 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:05.813761 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:05.813766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:05.813977 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 04:36:05.814129 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 04:36:05.814357 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260924 04:36:05.816120 3912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44569
I20260924 04:36:05.816320 3912 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:05.816736 3912 heartbeater.cc:511] Master 127.0.98.62:44569 requested a full tablet report, sending...
I20260924 04:36:05.817647 3453 ts_manager.cc:194] Registered new tserver with Master: 74b40d0ce3b942dbba0b20f9e85bcc8b (127.0.98.3:33049)
I20260924 04:36:05.818163 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45717
I20260924 04:36:05.819708 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224565, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224565, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:05.844214 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:05.835935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.0.98.62:44569)
0924 04:36:05.836068 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:05.836072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:05.836264 (+ 192us) server_negotiation.cc:407] Connection header received
0924 04:36:05.836388 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:05.836391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:05.836433 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:05.836496 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:05.837173 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.837707 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.838537 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:05.838755 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:05.838841 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:05.841123 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:05.841127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:05.841129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:05.841193 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:05.842577 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.842976 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.842981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.842983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.843076 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:05.843334 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:05.843338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:05.843340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:05.843554 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:05.843687 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:05.843923 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 04:36:05.844040 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 04:36:05.844573 3453 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51644
I20260924 04:36:05.845297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 3515
I20260924 04:36:05.850898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 3649
I20260924 04:36:05.856125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 3783
I20260924 04:36:05.860867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 3421
2026-09-24T04:36:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2132 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3924](info): set up 2 sockets
Sep 24 04:36:05 dist-test-slave-jlnt krb5kdc[3924](info): commencing operation
krb5kdc: starting...
W20260924 04:36:06.901223 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:06 dist-test-slave-jlnt krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:36:07.020432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:07.129712 3937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:07.129962 3937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:07.130002 3937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:07.133829 3937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:07.133875 3937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:07.133894 3937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:07.133911 3937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:07.133926 3937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:07.138995 3937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:07.139899 3937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:07.140851 3937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:07.148907 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:07.149012 3937 server_base.cc:1034] running on GCE node
W20260924 04:36:07.148940 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:07.148939 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:07.149852 3937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:07.150882 3937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:07.152050 3937 hybrid_clock.cc:648] HybridClock initialized: now 1790224567152024 us; error 42 us; skew 500 ppm
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:07.154731 3937 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:36:07.155589 3937 webserver.cc:460] Webserver started at http://127.0.98.62:42737/ using document root <none> and password file <none>
I20260924 04:36:07.156162 3937 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:07.156245 3937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:07.156466 3937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:07.158480 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "cc6c885f4717479e9241aeb4b49fab47"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "95e25373d67f8f9d4cb725b5e1f41b3f"
server_key_iv: "35200a6ef2c1bb4b0b805dc5449a9e78"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.159029 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc6c885f4717479e9241aeb4b49fab47"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "95e25373d67f8f9d4cb725b5e1f41b3f"
server_key_iv: "35200a6ef2c1bb4b0b805dc5449a9e78"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.162544 3937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:07.164783 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.166026 3937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:07.166148 3937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "cc6c885f4717479e9241aeb4b49fab47"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "95e25373d67f8f9d4cb725b5e1f41b3f"
server_key_iv: "35200a6ef2c1bb4b0b805dc5449a9e78"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.166249 3937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:07.195006 3937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:07.198145 3937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:07.198351 3937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:07.205370 3937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42549
I20260924 04:36:07.205389 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42549 every 8 connection(s)
I20260924 04:36:07.206334 3937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:07.206689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 3937
I20260924 04:36:07.206830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:07.207113 392 external_mini_cluster.cc:1428] Setting key bfc87959fc55a5b7669d0f9fcbde3115
I20260924 04:36:07.209723 4005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:07.216128 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47: Bootstrap starting.
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:07.218864 4005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:07.219715 4005 log.cc:824] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:07.221844 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47: No bootstrap required, opened a new log
I20260924 04:36:07.222069 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:07.208487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.0.98.62:42549)
0924 04:36:07.208914 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:07.208933 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:07.208976 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:36:07.209869 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:07.209892 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:07.210302 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:07.210651 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:07.211321 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.212824 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.213739 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.214416 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.214587 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:07.217167 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:07.217176 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:07.217196 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:07.217231 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:07.219142 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.219525 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.219531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.219536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.219674 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.219926 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.219929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.219931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.220366 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:07.220578 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:07.220834 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 04:36:07.221121 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":117,"threads_started":1}
I20260924 04:36:07.225582 4005 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6c885f4717479e9241aeb4b49fab47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260924 04:36:07.225785 4005 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:07.225819 4005 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6c885f4717479e9241aeb4b49fab47, State: Initialized, Role: FOLLOWER
I20260924 04:36:07.226195 4005 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6c885f4717479e9241aeb4b49fab47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260924 04:36:07.226311 4005 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:07.226356 4005 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:07.226424 4005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:07.227212 4005 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6c885f4717479e9241aeb4b49fab47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260924 04:36:07.227555 4005 leader_election.cc:302] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc6c885f4717479e9241aeb4b49fab47; no voters:
I20260924 04:36:07.227905 4005 leader_election.cc:288] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:07.227978 4010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:07.228266 4010 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [term 1 LEADER]: Becoming Leader. State: Replica: cc6c885f4717479e9241aeb4b49fab47, State: Running, Role: LEADER
I20260924 04:36:07.228577 4010 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6c885f4717479e9241aeb4b49fab47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } }
I20260924 04:36:07.229272 4005 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:07.230445 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc6c885f4717479e9241aeb4b49fab47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6c885f4717479e9241aeb4b49fab47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } } }
I20260924 04:36:07.230571 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:07.230808 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc6c885f4717479e9241aeb4b49fab47. Latest consensus state: current_term: 1 leader_uuid: "cc6c885f4717479e9241aeb4b49fab47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6c885f4717479e9241aeb4b49fab47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42549 } } }
I20260924 04:36:07.231372 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:07.231402 4019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:07.234277 4019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:07.239459 4019 catalog_manager.cc:1347] Generated new cluster ID: 20c4de594f1c49fd9f7b5dcec6d3691d
I20260924 04:36:07.239518 4019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:07.245887 4019 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:07.246984 4019 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:07.258080 4019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc6c885f4717479e9241aeb4b49fab47: Generated new TSK 0
I20260924 04:36:07.258842 4019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:36:07.290045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:07.398640 4031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:07.399003 4031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:07.399139 4031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:07.403671 4031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:07.403759 4031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:07.403858 4031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:07.409510 4031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:07.410497 4031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:07.411489 4031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:07.420578 4039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:07.420625 4037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:07.420753 4031 server_base.cc:1034] running on GCE node
W20260924 04:36:07.421007 4036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:07.421479 4031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:07.422075 4031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:07.423228 4031 hybrid_clock.cc:648] HybridClock initialized: now 1790224567423211 us; error 29 us; skew 500 ppm
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:07.425864 4031 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:36:07.426653 4031 webserver.cc:460] Webserver started at http://127.0.98.1:42501/ using document root <none> and password file <none>
I20260924 04:36:07.427192 4031 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:07.427251 4031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:07.427480 4031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:07.429445 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "3a7c68b6156947e79cfae162d6239c10"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "cb933e2a9f2fc2fadf8573fa41fea26e"
server_key_iv: "b2a428b83e290c410924522c0141bbe3"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.430023 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3a7c68b6156947e79cfae162d6239c10"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "cb933e2a9f2fc2fadf8573fa41fea26e"
server_key_iv: "b2a428b83e290c410924522c0141bbe3"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.433631 4031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:36:07.436070 4046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.437386 4031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:07.437496 4031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "3a7c68b6156947e79cfae162d6239c10"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "cb933e2a9f2fc2fadf8573fa41fea26e"
server_key_iv: "b2a428b83e290c410924522c0141bbe3"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.437597 4031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:07.461671 4031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:07.464707 4031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:07.465008 4031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:07.465617 4031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:07.466439 4031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:07.466485 4031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.466523 4031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:07.466545 4031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.475272 4031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36263
I20260924 04:36:07.475299 4159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36263 every 8 connection(s)
I20260924 04:36:07.476200 4031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:07.486140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 4031
I20260924 04:36:07.486253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:07.486524 392 external_mini_cluster.cc:1428] Setting key e1b91400b505e8d0f5af59d06bd48844
I20260924 04:36:07.489466 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:07.477876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39463 (local address 127.0.98.62:42549)
0924 04:36:07.478002 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:07.478007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:07.478818 (+ 811us) server_negotiation.cc:407] Connection header received
0924 04:36:07.480090 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:07.480092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:07.480146 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:07.480218 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:07.481408 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.481948 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.483006 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.483240 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.483322 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:07.485636 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:07.485639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:07.485641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:07.485663 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:07.487361 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.487879 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.487883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.487885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.487966 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.488323 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.488325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.488326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.488480 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:07.488549 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:07.489216 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 04:36:07.489324 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:36:07.490398 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:07.478251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42549 (local address 127.0.98.1:39463)
0924 04:36:07.478665 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:07.478720 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:07.479906 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:07.480374 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:07.480380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:07.480732 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:07.481251 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:07.481265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.482058 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.482061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:07.482907 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:07.482915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.483393 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.483396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:07.483535 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:07.484084 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:07.484100 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:07.485498 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:07.487463 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:07.487467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:07.487477 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:07.487783 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:07.488058 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:07.488061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:07.488063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:07.488227 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:07.488575 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:07.488585 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:07.488895 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 04:36:07.489096 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 04:36:07.489453 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260924 04:36:07.491506 4160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42549
I20260924 04:36:07.491744 4160 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:07.492312 4160 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260924 04:36:07.493942 3969 ts_manager.cc:194] Registered new tserver with Master: 3a7c68b6156947e79cfae162d6239c10 (127.0.98.1:36263)
I20260924 04:36:07.495525 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39463
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:36:07.516069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:07.624653 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:07.624934 4165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:07.625010 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:07.629094 4165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:07.629186 4165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:07.629271 4165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:07.634369 4165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:07.635341 4165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:07.636308 4165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:07.646459 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:07.646461 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:07.646461 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:07.646687 4165 server_base.cc:1034] running on GCE node
I20260924 04:36:07.647109 4165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:07.647742 4165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:07.648926 4165 hybrid_clock.cc:648] HybridClock initialized: now 1790224567648907 us; error 38 us; skew 500 ppm
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:07.651643 4165 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:36:07.652438 4165 webserver.cc:460] Webserver started at http://127.0.98.2:35831/ using document root <none> and password file <none>
I20260924 04:36:07.652936 4165 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:07.652992 4165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:07.653218 4165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:07.655224 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ce1d86f8cd84b148d620878f95b51bf"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "8a59372a552b5f4db1db1c5285787b7a"
server_key_iv: "49bde3d64a0801ef1b495b0c59ca0a4a"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.655766 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ce1d86f8cd84b148d620878f95b51bf"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "8a59372a552b5f4db1db1c5285787b7a"
server_key_iv: "49bde3d64a0801ef1b495b0c59ca0a4a"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.659387 4165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:07.661712 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.662806 4165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:07.662918 4165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "0ce1d86f8cd84b148d620878f95b51bf"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "8a59372a552b5f4db1db1c5285787b7a"
server_key_iv: "49bde3d64a0801ef1b495b0c59ca0a4a"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.663012 4165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:07.691916 4165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:07.694839 4165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:07.695029 4165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:07.695608 4165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:07.696393 4165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:07.696439 4165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.696502 4165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:07.696540 4165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.705295 4165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36613
I20260924 04:36:07.705322 4293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36613 every 8 connection(s)
I20260924 04:36:07.706166 4165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:07.712288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 4165
I20260924 04:36:07.712421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:07.712714 392 external_mini_cluster.cc:1428] Setting key a0731d007f0175679bf13678af525150
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:07.719599 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:07.707910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50035 (local address 127.0.98.62:42549)
0924 04:36:07.708061 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:07.708065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:07.708880 (+ 815us) server_negotiation.cc:407] Connection header received
0924 04:36:07.710259 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:07.710261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:07.710308 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:07.710371 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:07.711467 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.711977 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.713223 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.713525 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.713662 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:07.716072 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:07.716079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:07.716082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:07.716113 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:07.717589 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.718172 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.718176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.718178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.718255 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.718569 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.718574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.718576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.718759 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:07.718859 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:07.719328 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 04:36:07.719442 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:36:07.720424 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:07.708294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42549 (local address 127.0.98.2:50035)
0924 04:36:07.708726 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:07.708776 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:07.710060 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:07.710534 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:07.710541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:07.710876 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:07.711348 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:07.711361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.712072 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.712075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:07.713078 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:07.713089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.713672 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.713676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:07.713875 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:07.714508 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:07.714523 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:07.715908 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:07.717716 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:07.717723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:07.717737 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:07.718060 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:07.718346 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:07.718349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:07.718351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:07.718482 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:07.718857 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:07.718863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:07.719101 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 04:36:07.719288 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 04:36:07.719594 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260924 04:36:07.721592 4294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42549
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:36:07.721830 4294 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:07.722357 4294 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260924 04:36:07.723269 3969 ts_manager.cc:194] Registered new tserver with Master: 0ce1d86f8cd84b148d620878f95b51bf (127.0.98.2:36613)
I20260924 04:36:07.723809 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:36:07.740141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:07.848568 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:07.848807 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:07.848848 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:07.852823 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:07.852876 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:07.852932 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:07.858566 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:07.859511 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:07.860433 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:07.869640 4307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:07.869653 4305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:07.869642 4304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:07.869686 4299 server_base.cc:1034] running on GCE node
I20260924 04:36:07.870337 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:07.870965 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:07.872146 4299 hybrid_clock.cc:648] HybridClock initialized: now 1790224567872133 us; error 25 us; skew 500 ppm
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:07.874806 4299 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:36:07.875661 4299 webserver.cc:460] Webserver started at http://127.0.98.3:35249/ using document root <none> and password file <none>
I20260924 04:36:07.876206 4299 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:07.876292 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:07.876516 4299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:07.878551 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "98b267d0e98c4bef8d7f3502a4cb75c9"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "d7ac57dce031e6f32b4081c3b2419b22"
server_key_iv: "cd848a8810a18d0572a56324bbe24b30"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.879112 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "98b267d0e98c4bef8d7f3502a4cb75c9"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "d7ac57dce031e6f32b4081c3b2419b22"
server_key_iv: "cd848a8810a18d0572a56324bbe24b30"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.882671 4299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:36:07.884950 4314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.886152 4299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:36:07.886274 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "98b267d0e98c4bef8d7f3502a4cb75c9"
format_stamp: "Formatted at 2026-09-24 04:36:07 on dist-test-slave-jlnt"
server_key: "d7ac57dce031e6f32b4081c3b2419b22"
server_key_iv: "cd848a8810a18d0572a56324bbe24b30"
server_key_version: "encryptionkey@0"
I20260924 04:36:07.886375 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:07.905939 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:07.908679 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:07.908869 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:07.909446 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:07.910243 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:07.910310 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.910399 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:07.910446 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:07.919287 4299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43705
I20260924 04:36:07.919307 4427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43705 every 8 connection(s)
I20260924 04:36:07.920172 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:07.925858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 4299
I20260924 04:36:07.926009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:07.926297 392 external_mini_cluster.cc:1428] Setting key fd867df6ca1bccd9016aabe9986bb108
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:07.933610 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:07.921961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60663 (local address 127.0.98.62:42549)
0924 04:36:07.922102 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:07.922106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:07.922845 (+ 739us) server_negotiation.cc:407] Connection header received
0924 04:36:07.924239 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:07.924242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:07.924303 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:07.924404 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:07.925604 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.926227 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.927600 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.927828 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.927932 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:07.930421 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:07.930427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:07.930432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:07.930460 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:07.931697 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.932157 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.932162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.932164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.932240 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.932548 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.932553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.932555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.932731 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:07.932831 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:07.933319 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 04:36:07.933433 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:36:07.934363 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:07.922289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42549 (local address 127.0.98.3:60663)
0924 04:36:07.922706 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:07.922745 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:07.924022 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:07.924574 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:07.924581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:07.924927 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:07.925459 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:07.925474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.926340 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.926348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:07.927478 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:07.927488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.927965 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.927969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:07.928191 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:07.928739 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:07.928753 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:07.930273 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:07.931801 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:07.931806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:07.931819 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:07.932054 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:07.932326 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:07.932330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:07.932332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:07.932461 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:07.932827 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:07.932833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:07.933067 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 04:36:07.933275 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 04:36:07.933569 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260924 04:36:07.935336 4428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42549
I20260924 04:36:07.935554 4428 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:07.936017 4428 heartbeater.cc:511] Master 127.0.98.62:42549 requested a full tablet report, sending...
I20260924 04:36:07.936942 3969 ts_manager.cc:194] Registered new tserver with Master: 98b267d0e98c4bef8d7f3502a4cb75c9 (127.0.98.3:43705)
I20260924 04:36:07.937515 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60663
I20260924 04:36:07.939708 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:36:07.988902 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:07.979201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.0.98.1:36263)
0924 04:36:07.979537 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:07.979554 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:07.979581 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:36:07.979632 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:07.979637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:07.979831 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:07.979956 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:07.980667 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.981564 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.982467 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.982874 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.982960 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:07.985310 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:07.985318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:07.985332 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:07.985357 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:07.987043 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.987433 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.987438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.987439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.987513 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.987826 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.987828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.987829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.988053 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:07.988377 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:07.988513 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 04:36:07.988699 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
Sep 24 04:36:07 dist-test-slave-jlnt krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:07.999411 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:07.991878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.0.98.62:42549)
0924 04:36:07.992027 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:07.992031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:07.992045 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:36:07.992142 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:07.992144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:07.992185 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:07.992250 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:07.992788 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.993356 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.994140 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:07.994341 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:07.994417 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:07.996560 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:07.996564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:07.996566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:07.996594 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:07.997845 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.998236 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.998241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.998243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.998317 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:07.998574 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:07.998577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:07.998579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:07.998778 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:07.998915 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:07.999141 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:36:07.999257 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 04:36:07.999799 3969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36238
I20260924 04:36:08.000347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 4031
I20260924 04:36:08.005785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 4165
I20260924 04:36:08.011075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 4299
I20260924 04:36:08.016038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 3937
2026-09-24T04:36:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2155 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:08 dist-test-slave-jlnt krb5kdc[4440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:08 dist-test-slave-jlnt krb5kdc[4440](info): set up 2 sockets
Sep 24 04:36:08 dist-test-slave-jlnt krb5kdc[4440](info): commencing operation
krb5kdc: starting...
W20260924 04:36:09.045183 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:09 dist-test-slave-jlnt krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:36:09.164505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:09.274802 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:09.275077 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:09.275120 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:09.279094 4453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:09.279150 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:09.279170 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:09.279186 4453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:09.279201 4453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:09.284264 4453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:09.285265 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:09.286221 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:09.294198 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:09.294183 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:09.294373 4453 server_base.cc:1034] running on GCE node
W20260924 04:36:09.294198 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:09.294932 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:09.295933 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:09.297116 4453 hybrid_clock.cc:648] HybridClock initialized: now 1790224569297089 us; error 40 us; skew 500 ppm
Sep 24 04:36:09 dist-test-slave-jlnt krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:09.299715 4453 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:36:09.300421 4453 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:09.300508 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:09.300724 4453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:09.302711 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "6b75cd7000d94e2eab8dc66219e553ac"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "1c44d1c406ea62fc78a67201eb46f294"
server_key_iv: "b776360444e537b20c18e48f9dfed527"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.303253 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b75cd7000d94e2eab8dc66219e553ac"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "1c44d1c406ea62fc78a67201eb46f294"
server_key_iv: "b776360444e537b20c18e48f9dfed527"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.306720 4453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:09.309226 4467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:09.310413 4453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:09.310539 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "6b75cd7000d94e2eab8dc66219e553ac"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "1c44d1c406ea62fc78a67201eb46f294"
server_key_iv: "b776360444e537b20c18e48f9dfed527"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.310647 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:09.327932 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:09.330789 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:09.330978 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:09.337931 4453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44517
I20260924 04:36:09.338192 4519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44517 every 8 connection(s)
I20260924 04:36:09.338616 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:09.340021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 4453
I20260924 04:36:09.340118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:09.340405 392 external_mini_cluster.cc:1428] Setting key 366efbee2cc048d6528c582bc16cd8be
I20260924 04:36:09.341475 4520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:09.347641 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac: Bootstrap starting.
Sep 24 04:36:09 dist-test-slave-jlnt krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:09.350147 4520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:09.350831 4520 log.cc:824] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:09.352762 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac: No bootstrap required, opened a new log
I20260924 04:36:09.354465 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:09.341758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.0.98.62:44517)
0924 04:36:09.342146 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:09.342164 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:09.342208 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:36:09.342828 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:09.342843 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:09.343159 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:09.343428 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:09.344311 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.345375 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.346543 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.346986 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.347100 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:09.349818 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:09.349823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:09.349835 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:09.349860 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:09.351764 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:09.352186 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:09.352191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:09.352197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:09.352307 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:09.352587 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:09.352589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:09.352591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:09.353019 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:09.353237 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:09.353520 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 04:36:09.353796 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":127,"threads_started":1}
I20260924 04:36:09.356336 4520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b75cd7000d94e2eab8dc66219e553ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } }
I20260924 04:36:09.356614 4520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:09.356689 4520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b75cd7000d94e2eab8dc66219e553ac, State: Initialized, Role: FOLLOWER
I20260924 04:36:09.357098 4520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b75cd7000d94e2eab8dc66219e553ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } }
I20260924 04:36:09.357286 4520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:09.357368 4520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:09.357465 4520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:09.358322 4520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b75cd7000d94e2eab8dc66219e553ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } }
I20260924 04:36:09.358644 4520 leader_election.cc:302] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b75cd7000d94e2eab8dc66219e553ac; no voters:
I20260924 04:36:09.358961 4520 leader_election.cc:288] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:09.359028 4525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:09.359236 4525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [term 1 LEADER]: Becoming Leader. State: Replica: 6b75cd7000d94e2eab8dc66219e553ac, State: Running, Role: LEADER
I20260924 04:36:09.359587 4525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b75cd7000d94e2eab8dc66219e553ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } }
I20260924 04:36:09.360196 4520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:09.361176 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b75cd7000d94e2eab8dc66219e553ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b75cd7000d94e2eab8dc66219e553ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } } }
I20260924 04:36:09.361404 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b75cd7000d94e2eab8dc66219e553ac. Latest consensus state: current_term: 1 leader_uuid: "6b75cd7000d94e2eab8dc66219e553ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b75cd7000d94e2eab8dc66219e553ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } } }
I20260924 04:36:09.361505 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:09.361586 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:09.362450 4535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:09.365497 4535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:09.370385 4535 catalog_manager.cc:1347] Generated new cluster ID: 13ca6255785b47eea6a320769932f54b
I20260924 04:36:09.370446 4535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:09.392652 4535 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:09.393716 4535 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:09.398556 4535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac: Generated new TSK 0
I20260924 04:36:09.399245 4535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:36:09.428359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44517
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:09.538687 4546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:09.538945 4546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:09.539019 4546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:09.542985 4546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:09.543165 4546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:09.543309 4546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:09.548718 4546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:09.549758 4546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:09.550755 4546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:09.559873 4551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:09.559911 4554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:09.560060 4546 server_base.cc:1034] running on GCE node
W20260924 04:36:09.559876 4552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:09.560662 4546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:09.561386 4546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:09.562575 4546 hybrid_clock.cc:648] HybridClock initialized: now 1790224569562556 us; error 39 us; skew 500 ppm
Sep 24 04:36:09 dist-test-slave-jlnt krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:09.565366 4546 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:36:09.566048 4546 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:09.566143 4546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:09.566354 4546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:09.568332 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "40f2b45e4571412f8b0c815ca46eaf90"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "2e16ac671c8bc6448980255203c5f6ef"
server_key_iv: "93277cd28dc42e784c8811a9a4dbc9ef"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.568885 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40f2b45e4571412f8b0c815ca46eaf90"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "2e16ac671c8bc6448980255203c5f6ef"
server_key_iv: "93277cd28dc42e784c8811a9a4dbc9ef"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.572387 4546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:36:09.574683 4560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:09.575799 4546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:09.575930 4546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "40f2b45e4571412f8b0c815ca46eaf90"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "2e16ac671c8bc6448980255203c5f6ef"
server_key_iv: "93277cd28dc42e784c8811a9a4dbc9ef"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.576032 4546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:09.596539 4546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:09.599875 4546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:09.600085 4546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:09.600659 4546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:09.601523 4546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:09.601590 4546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:09.601652 4546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:09.601698 4546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:09.610421 4546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32965
I20260924 04:36:09.610440 4673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32965 every 8 connection(s)
I20260924 04:36:09.610981 4546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:09.614528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 4546
I20260924 04:36:09.614665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:09.614985 392 external_mini_cluster.cc:1428] Setting key 043c864d36a1ec6ea3aa0f7829efdcc5
Sep 24 04:36:09 dist-test-slave-jlnt krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:36:09.625358 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:09.612575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44465 (local address 127.0.98.62:44517)
0924 04:36:09.612702 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:09.612707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:09.613522 (+ 815us) server_negotiation.cc:407] Connection header received
0924 04:36:09.615175 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:09.615178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:09.615226 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:09.615299 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:09.616669 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.617238 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.618659 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.618890 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.618993 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:09.621649 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:09.621656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:09.621662 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:09.621695 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:09.623332 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:09.623799 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:09.623803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:09.623804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:09.623893 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:09.624211 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:09.624215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:09.624219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:09.624453 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:09.624569 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:09.624995 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 04:36:09.625169 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:36:09.626120 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:09.612886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.98.1:44465)
0924 04:36:09.613374 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:09.613428 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:09.614963 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:09.615468 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:09.615476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:09.615904 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:09.616501 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:09.616521 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.617358 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.617364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:09.618526 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:09.618536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.619026 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.619030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:09.619253 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:09.619843 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:09.619868 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:09.621494 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:09.623442 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:09.623447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:09.623457 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:09.623705 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:09.623987 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:09.623989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:09.623990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:09.624119 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:09.624553 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:09.624558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:09.624767 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 04:36:09.624913 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 04:36:09.625255 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:36:09.627396 4674 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44517
I20260924 04:36:09.627615 4674 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:09.628026 4674 heartbeater.cc:511] Master 127.0.98.62:44517 requested a full tablet report, sending...
I20260924 04:36:09.629508 4484 ts_manager.cc:194] Registered new tserver with Master: 40f2b45e4571412f8b0c815ca46eaf90 (127.0.98.1:32965)
I20260924 04:36:09.630586 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:36:09.644248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44517
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:09.752686 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:09.752933 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:09.752977 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:09.757121 4679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:09.757220 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:09.757302 4679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:09.762742 4679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:09.763696 4679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:09.764652 4679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:09.773561 4687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:09.773561 4684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:09.773756 4679 server_base.cc:1034] running on GCE node
W20260924 04:36:09.773586 4685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:09.774271 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:09.774899 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:09.776070 4679 hybrid_clock.cc:648] HybridClock initialized: now 1790224569776045 us; error 41 us; skew 500 ppm
Sep 24 04:36:09 dist-test-slave-jlnt krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:09.778815 4679 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:36:09.779569 4679 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:09.779652 4679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:09.779902 4679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:09.781953 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "109cf9de82664b5cbc3b5fce06b251a5"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "6cee53c4beee2c623ee67e18f8712774"
server_key_iv: "ab492f54d008549090b37e37fc53c6d1"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.782430 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "109cf9de82664b5cbc3b5fce06b251a5"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "6cee53c4beee2c623ee67e18f8712774"
server_key_iv: "ab492f54d008549090b37e37fc53c6d1"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.785857 4679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:09.788049 4693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:09.789294 4679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:09.789389 4679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "109cf9de82664b5cbc3b5fce06b251a5"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "6cee53c4beee2c623ee67e18f8712774"
server_key_iv: "ab492f54d008549090b37e37fc53c6d1"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.789525 4679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:09.800942 4679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:09.803933 4679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:09.804127 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:09.804648 4679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:09.805464 4679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:09.805510 4679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:09.805572 4679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:09.805601 4679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:09.813683 4679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42621
I20260924 04:36:09.813755 4806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42621 every 8 connection(s)
I20260924 04:36:09.814286 4679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:09.820068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 4679
I20260924 04:36:09.820199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:09.820508 392 external_mini_cluster.cc:1428] Setting key 46c479ee94c4064814cc5432d25b0d5e
Sep 24 04:36:09 dist-test-slave-jlnt krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:09.827471 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:09.815930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39893 (local address 127.0.98.62:44517)
0924 04:36:09.816085 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:09.816090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:09.816864 (+ 774us) server_negotiation.cc:407] Connection header received
0924 04:36:09.818186 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:09.818189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:09.818235 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:09.818309 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:09.819404 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.819910 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.821376 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.821610 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.821728 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:09.824090 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:09.824094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:09.824097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:09.824125 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:09.825527 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:09.826036 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:09.826041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:09.826043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:09.826120 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:09.826432 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:09.826437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:09.826438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:09.826618 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:09.826719 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:09.827191 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 04:36:09.827303 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":57}
I20260924 04:36:09.828245 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:09.816311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.98.2:39893)
0924 04:36:09.816726 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:09.816766 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:09.817995 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:09.818462 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:09.818467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:09.818802 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:09.819278 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:09.819292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.820025 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.820028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:09.821258 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:09.821267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:09.821759 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:09.821762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:09.821925 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:09.822481 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:09.822497 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:09.823936 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:09.825672 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:09.825678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:09.825691 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:09.825937 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:09.826207 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:09.826212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:09.826214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:09.826344 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:09.826716 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:09.826723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:09.826954 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 04:36:09.827139 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 04:36:09.827448 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:36:09.829589 4807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44517
I20260924 04:36:09.829862 4807 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:09.830296 4807 heartbeater.cc:511] Master 127.0.98.62:44517 requested a full tablet report, sending...
I20260924 04:36:09.831312 4484 ts_manager.cc:194] Registered new tserver with Master: 109cf9de82664b5cbc3b5fce06b251a5 (127.0.98.2:42621)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:36:09.831939 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:36:09.847982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44517
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:09.956455 4812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:09.956722 4812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:09.956766 4812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:09.960834 4812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:09.960886 4812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:09.960983 4812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:09.966220 4812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:09.967207 4812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:09.968209 4812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:09.977329 4820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:09.977339 4817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:09.977465 4812 server_base.cc:1034] running on GCE node
W20260924 04:36:09.977358 4818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:09.978080 4812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:09.978621 4812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:09.979771 4812 hybrid_clock.cc:648] HybridClock initialized: now 1790224569979753 us; error 29 us; skew 500 ppm
Sep 24 04:36:09 dist-test-slave-jlnt krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:09.982613 4812 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:36:09.983362 4812 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:09.983424 4812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:09.983666 4812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:09.985723 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "07957d3f567240e3981938a01b06ab59"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "f1ce5915250eab042f254ca9f669a424"
server_key_iv: "ac5d34216c340ee281ec77eee516872e"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.986275 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "07957d3f567240e3981938a01b06ab59"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "f1ce5915250eab042f254ca9f669a424"
server_key_iv: "ac5d34216c340ee281ec77eee516872e"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.989867 4812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:36:09.992200 4826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:09.993453 4812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 04:36:09.993593 4812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "07957d3f567240e3981938a01b06ab59"
format_stamp: "Formatted at 2026-09-24 04:36:09 on dist-test-slave-jlnt"
server_key: "f1ce5915250eab042f254ca9f669a424"
server_key_iv: "ac5d34216c340ee281ec77eee516872e"
server_key_version: "encryptionkey@0"
I20260924 04:36:09.993686 4812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:10.009253 4812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:10.012368 4812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:10.012595 4812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:10.013303 4812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:10.014245 4812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:10.014297 4812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.014338 4812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:10.014356 4812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.023309 4812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32875
I20260924 04:36:10.023331 4939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32875 every 8 connection(s)
I20260924 04:36:10.024036 4812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:36:10 dist-test-slave-jlnt krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:10.033707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 4812
I20260924 04:36:10.033818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:10.034128 392 external_mini_cluster.cc:1428] Setting key dbe4733f0f24812e050f6683dc438e0e
I20260924 04:36:10.037603 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.025886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47299 (local address 127.0.98.62:44517)
0924 04:36:10.026021 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.026024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.026835 (+ 811us) server_negotiation.cc:407] Connection header received
0924 04:36:10.028167 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.028170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.028227 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.028324 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:10.029534 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.030062 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.031176 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.031394 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.031499 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.034015 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:10.034018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:10.034020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:10.034046 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:10.035561 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:10.036100 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:10.036104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:10.036106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:10.036186 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:10.036525 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:10.036527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:10.036528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:10.036694 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:10.036767 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.037236 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.037329 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:36:10.038234 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.026267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.98.3:47299)
0924 04:36:10.026695 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.026736 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.027957 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.028498 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.028506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.028857 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:10.029387 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.029402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.030192 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.030195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.031049 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.031057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.031536 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.031540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.031704 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.032265 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:10.032280 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:10.033847 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:10.035666 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:10.035672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:10.035685 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:10.036011 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:10.036263 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:10.036265 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:10.036267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:10.036428 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:10.036773 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:10.036777 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:10.036987 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.037172 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.037430 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260924 04:36:10.039259 4940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44517
I20260924 04:36:10.039500 4940 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:10.039911 4940 heartbeater.cc:511] Master 127.0.98.62:44517 requested a full tablet report, sending...
I20260924 04:36:10.040805 4484 ts_manager.cc:194] Registered new tserver with Master: 07957d3f567240e3981938a01b06ab59 (127.0.98.3:32875)
I20260924 04:36:10.041368 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47299
I20260924 04:36:10.047816 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:36:10.056409 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.049684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.0.98.62:44517)
0924 04:36:10.049792 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.049796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.049986 (+ 190us) server_negotiation.cc:407] Connection header received
0924 04:36:10.050064 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.050066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.050103 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.050181 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:10.050816 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.051318 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.052105 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.052307 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.052387 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.053711 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:10.053715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:10.053717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:10.053738 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:10.055019 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:10.055403 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:10.055406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:10.055407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:10.055478 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:10.055741 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:10.055743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:10.055744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:10.055936 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:10.056039 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.056215 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.056299 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:36:10.058959 4484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:36:10.062139 4484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:10.076498 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.070397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42621 (local address 127.0.0.1:54734)
0924 04:36:10.070715 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.070752 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.070852 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.071826 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.071832 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.071854 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:10.071990 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.071998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.073597 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.073600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.074980 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.074989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.076012 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.076014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.076089 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.076118 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.076240 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.076354 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260924 04:36:10.076525 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.070593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.0.98.2:42621)
0924 04:36:10.070987 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.071000 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.071026 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:36:10.071110 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.071116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.071339 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.071498 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:10.072217 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.073473 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.075094 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.075911 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.076015 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.076161 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.076280 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.076372 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":101,"threads_started":1}
I20260924 04:36:10.077726 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.071146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32965 (local address 127.0.0.1:47458)
0924 04:36:10.071723 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.071738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.071820 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.072502 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.072505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.072523 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:10.072621 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.072627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.074099 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.074102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.076118 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.076127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.077190 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.077193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.077324 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.077424 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.077487 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.077596 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":1024,"thread_start_us":87,"threads_started":1}
I20260924 04:36:10.077996 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.071913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.0.98.3:32875)
0924 04:36:10.072414 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.072425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.072447 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:36:10.072517 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.072522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.072700 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.072850 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:10.073030 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.073977 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.076291 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.077012 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.077568 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.077701 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.077750 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.077839 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":154,"threads_started":1}
I20260924 04:36:10.078279 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.071341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.98.1:32965)
0924 04:36:10.072073 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.072088 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.072113 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:36:10.072188 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.072193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.072365 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.072526 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:10.072719 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.073830 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.076243 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.077021 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.077137 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.078002 (+ 865us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.078050 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.078136 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":89,"threads_started":1}
I20260924 04:36:10.078307 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.071457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32875 (local address 127.0.0.1:36888)
0924 04:36:10.071976 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.071988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.072170 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.072841 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.072844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.072861 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:10.072937 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.072942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.074138 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.074143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.075102 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.075112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.077175 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.077178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.077320 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.077421 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.077487 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.078177 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":642,"thread_start_us":86,"threads_started":1}
I20260924 04:36:10.080070 4741 tablet_service.cc:1467] Processing CreateTablet for tablet 51a3ef5da8084f2babde5afd7753f84c (DEFAULT_TABLE table=test-table [id=b5236fe15d4c4d7ea64b26cc67a10612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:10.080637 4608 tablet_service.cc:1467] Processing CreateTablet for tablet 51a3ef5da8084f2babde5afd7753f84c (DEFAULT_TABLE table=test-table [id=b5236fe15d4c4d7ea64b26cc67a10612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:10.081213 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a3ef5da8084f2babde5afd7753f84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:10.080945 4874 tablet_service.cc:1467] Processing CreateTablet for tablet 51a3ef5da8084f2babde5afd7753f84c (DEFAULT_TABLE table=test-table [id=b5236fe15d4c4d7ea64b26cc67a10612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:10.081586 4608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a3ef5da8084f2babde5afd7753f84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:10.081928 4874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a3ef5da8084f2babde5afd7753f84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:10.087173 4957 tablet_bootstrap.cc:492] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90: Bootstrap starting.
I20260924 04:36:10.088495 4958 tablet_bootstrap.cc:492] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: Bootstrap starting.
I20260924 04:36:10.089067 4959 tablet_bootstrap.cc:492] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: Bootstrap starting.
I20260924 04:36:10.089494 4957 tablet_bootstrap.cc:654] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:10.090744 4958 tablet_bootstrap.cc:654] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:10.090920 4957 log.cc:824] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:10.091166 4959 tablet_bootstrap.cc:654] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:10.092134 4958 log.cc:824] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:10.092435 4959 log.cc:824] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:10.093036 4957 tablet_bootstrap.cc:492] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90: No bootstrap required, opened a new log
I20260924 04:36:10.093245 4957 ts_tablet_manager.cc:1397] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:36:10.094023 4958 tablet_bootstrap.cc:492] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: No bootstrap required, opened a new log
I20260924 04:36:10.094213 4958 ts_tablet_manager.cc:1397] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:36:10.097199 4957 raft_consensus.cc:348] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.097479 4957 raft_consensus.cc:374] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:10.097586 4957 raft_consensus.cc:729] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f2b45e4571412f8b0c815ca46eaf90, State: Initialized, Role: FOLLOWER
I20260924 04:36:10.098042 4957 consensus_queue.cc:260] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.098031 4958 raft_consensus.cc:348] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.098322 4958 raft_consensus.cc:374] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:10.098387 4958 raft_consensus.cc:729] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109cf9de82664b5cbc3b5fce06b251a5, State: Initialized, Role: FOLLOWER
I20260924 04:36:10.098742 4959 tablet_bootstrap.cc:492] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: No bootstrap required, opened a new log
I20260924 04:36:10.098780 4958 consensus_queue.cc:260] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.098946 4959 ts_tablet_manager.cc:1397] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260924 04:36:10.099058 4957 ts_tablet_manager.cc:1428] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:36:10.099426 4674 heartbeater.cc:503] Master 127.0.98.62:44517 was elected leader, sending a full tablet report...
I20260924 04:36:10.099431 4807 heartbeater.cc:503] Master 127.0.98.62:44517 was elected leader, sending a full tablet report...
I20260924 04:36:10.099709 4958 ts_tablet_manager.cc:1428] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:36:10.102867 4959 raft_consensus.cc:348] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.103130 4959 raft_consensus.cc:374] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:10.103192 4959 raft_consensus.cc:729] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07957d3f567240e3981938a01b06ab59, State: Initialized, Role: FOLLOWER
I20260924 04:36:10.103622 4959 consensus_queue.cc:260] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.104310 4940 heartbeater.cc:503] Master 127.0.98.62:44517 was elected leader, sending a full tablet report...
I20260924 04:36:10.104562 4959 ts_tablet_manager.cc:1428] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
W20260924 04:36:10.111932 4675 tablet.cc:2369] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:36:10.148098 4963 raft_consensus.cc:482] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:10.148280 4963 raft_consensus.cc:504] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.149452 4963 leader_election.cc:288] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 109cf9de82664b5cbc3b5fce06b251a5 (127.0.98.2:42621), 07957d3f567240e3981938a01b06ab59 (127.0.98.3:32875)
I20260924 04:36:10.150527 4965 raft_consensus.cc:482] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:10.150806 4965 raft_consensus.cc:504] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.151961 4965 leader_election.cc:288] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 109cf9de82664b5cbc3b5fce06b251a5 (127.0.98.2:42621), 40f2b45e4571412f8b0c815ca46eaf90 (127.0.98.1:32965)
I20260924 04:36:10.154284 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.149719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42621 (local address 127.0.98.1:50437)
0924 04:36:10.150156 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.150172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.150313 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.150622 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.150625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.150649 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:10.150777 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.150784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.151604 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.151608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.152839 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.152848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.153841 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.153845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.153944 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.153967 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.154036 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.154106 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":36,"thread_start_us":114,"threads_started":1}
I20260924 04:36:10.154834 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.149833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50437 (local address 127.0.98.2:42621)
0924 04:36:10.149983 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.149989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.150231 (+ 242us) server_negotiation.cc:407] Connection header received
0924 04:36:10.150462 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.150468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.150533 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.150605 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:10.150897 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.151498 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.152979 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.153635 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.153729 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.153786 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.154625 (+ 839us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.154718 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:36:10.154973 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.150233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39191 (local address 127.0.98.3:32875)
0924 04:36:10.150361 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.150367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.150539 (+ 172us) server_negotiation.cc:407] Connection header received
0924 04:36:10.150674 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.150678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.150730 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.150775 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:10.151203 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.151893 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.153485 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.154114 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.154184 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.154233 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.154737 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.154852 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":54}
I20260924 04:36:10.154994 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.150137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32875 (local address 127.0.98.1:39191)
0924 04:36:10.150449 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.150462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.150546 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.150957 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.150960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.150977 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:10.151075 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.151081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.152027 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.152030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.153349 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.153359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.154327 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.154330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.154457 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.154625 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.154784 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.154848 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":172,"thread_start_us":105,"threads_started":1}
I20260924 04:36:10.155370 4761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a3ef5da8084f2babde5afd7753f84c" candidate_uuid: "40f2b45e4571412f8b0c815ca46eaf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "109cf9de82664b5cbc3b5fce06b251a5" is_pre_election: true
I20260924 04:36:10.155678 4761 raft_consensus.cc:2454] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40f2b45e4571412f8b0c815ca46eaf90 in term 0.
I20260924 04:36:10.155830 4894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a3ef5da8084f2babde5afd7753f84c" candidate_uuid: "40f2b45e4571412f8b0c815ca46eaf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07957d3f567240e3981938a01b06ab59" is_pre_election: true
I20260924 04:36:10.155998 4894 raft_consensus.cc:2454] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40f2b45e4571412f8b0c815ca46eaf90 in term 0.
I20260924 04:36:10.156234 4562 leader_election.cc:302] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 109cf9de82664b5cbc3b5fce06b251a5, 40f2b45e4571412f8b0c815ca46eaf90; no voters:
I20260924 04:36:10.156620 4963 raft_consensus.cc:2781] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:10.156702 4963 raft_consensus.cc:482] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:10.156752 4963 raft_consensus.cc:3037] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:10.156903 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.152109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42621 (local address 127.0.98.3:49499)
0924 04:36:10.152210 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.152225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.152297 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.153126 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.153132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.153174 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:10.153286 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.153296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.154394 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.154397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.155558 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.155568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.156498 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.156501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.156599 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.156633 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.156713 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.156779 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 04:36:10.157373 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.152525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49499 (local address 127.0.98.2:42621)
0924 04:36:10.152860 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.152868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.152883 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:10.152925 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.152929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.152976 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.153386 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:10.153478 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.154281 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.155759 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.156383 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.156488 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.156545 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.157172 (+ 627us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.157248 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260924 04:36:10.157478 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.152330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32965 (local address 127.0.98.3:56411)
0924 04:36:10.153071 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.153085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.153232 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.153559 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.153562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.153577 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:10.153676 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.153682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.154632 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.154635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.156075 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:10.156083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.157046 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.157049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:10.157178 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.157198 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.157262 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.157320 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":98,"threads_started":1}
I20260924 04:36:10.157478 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.152826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56411 (local address 127.0.98.1:32965)
0924 04:36:10.152913 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.152917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.153178 (+ 261us) server_negotiation.cc:407] Connection header received
0924 04:36:10.153343 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.153347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.153392 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.153434 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:10.153794 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.154516 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.156221 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.156803 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.156869 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.156919 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.157299 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.157371 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":34}
I20260924 04:36:10.157923 4761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a3ef5da8084f2babde5afd7753f84c" candidate_uuid: "07957d3f567240e3981938a01b06ab59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "109cf9de82664b5cbc3b5fce06b251a5" is_pre_election: true
I20260924 04:36:10.158063 4761 raft_consensus.cc:2454] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07957d3f567240e3981938a01b06ab59 in term 0.
I20260924 04:36:10.158056 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a3ef5da8084f2babde5afd7753f84c" candidate_uuid: "07957d3f567240e3981938a01b06ab59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40f2b45e4571412f8b0c815ca46eaf90" is_pre_election: true
I20260924 04:36:10.158140 4963 raft_consensus.cc:504] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.158373 4628 raft_consensus.cc:2379] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 07957d3f567240e3981938a01b06ab59 in current term 1: Already voted for candidate 40f2b45e4571412f8b0c815ca46eaf90 in this term.
I20260924 04:36:10.158408 4828 leader_election.cc:302] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07957d3f567240e3981938a01b06ab59, 109cf9de82664b5cbc3b5fce06b251a5; no voters:
I20260924 04:36:10.158507 4963 leader_election.cc:288] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [CANDIDATE]: Term 1 election: Requested vote from peers 109cf9de82664b5cbc3b5fce06b251a5 (127.0.98.2:42621), 07957d3f567240e3981938a01b06ab59 (127.0.98.3:32875)
I20260924 04:36:10.158655 4965 raft_consensus.cc:2781] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:10.158752 4965 raft_consensus.cc:482] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:10.158847 4965 raft_consensus.cc:3037] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:10.159058 4894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a3ef5da8084f2babde5afd7753f84c" candidate_uuid: "40f2b45e4571412f8b0c815ca46eaf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07957d3f567240e3981938a01b06ab59"
I20260924 04:36:10.159155 4761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a3ef5da8084f2babde5afd7753f84c" candidate_uuid: "40f2b45e4571412f8b0c815ca46eaf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "109cf9de82664b5cbc3b5fce06b251a5"
I20260924 04:36:10.159237 4761 raft_consensus.cc:3037] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:10.160089 4965 raft_consensus.cc:504] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.160317 4894 raft_consensus.cc:2379] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 40f2b45e4571412f8b0c815ca46eaf90 in current term 1: Already voted for candidate 07957d3f567240e3981938a01b06ab59 in this term.
I20260924 04:36:10.160296 4761 raft_consensus.cc:2454] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40f2b45e4571412f8b0c815ca46eaf90 in term 1.
I20260924 04:36:10.160394 4965 leader_election.cc:288] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [CANDIDATE]: Term 1 election: Requested vote from peers 109cf9de82664b5cbc3b5fce06b251a5 (127.0.98.2:42621), 40f2b45e4571412f8b0c815ca46eaf90 (127.0.98.1:32965)
I20260924 04:36:10.160794 4562 leader_election.cc:302] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 109cf9de82664b5cbc3b5fce06b251a5, 40f2b45e4571412f8b0c815ca46eaf90; no voters:
I20260924 04:36:10.160967 4963 raft_consensus.cc:2781] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:10.161197 4963 raft_consensus.cc:686] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 1 LEADER]: Becoming Leader. State: Replica: 40f2b45e4571412f8b0c815ca46eaf90, State: Running, Role: LEADER
I20260924 04:36:10.161176 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a3ef5da8084f2babde5afd7753f84c" candidate_uuid: "07957d3f567240e3981938a01b06ab59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40f2b45e4571412f8b0c815ca46eaf90"
I20260924 04:36:10.161551 4963 consensus_queue.cc:237] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } }
I20260924 04:36:10.162969 4761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a3ef5da8084f2babde5afd7753f84c" candidate_uuid: "07957d3f567240e3981938a01b06ab59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "109cf9de82664b5cbc3b5fce06b251a5"
I20260924 04:36:10.163146 4761 raft_consensus.cc:2379] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 07957d3f567240e3981938a01b06ab59 in current term 1: Already voted for candidate 40f2b45e4571412f8b0c815ca46eaf90 in this term.
I20260924 04:36:10.163538 4828 leader_election.cc:302] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07957d3f567240e3981938a01b06ab59; no voters: 109cf9de82664b5cbc3b5fce06b251a5, 40f2b45e4571412f8b0c815ca46eaf90
I20260924 04:36:10.163756 4965 raft_consensus.cc:2726] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 04:36:10.164501 4484 catalog_manager.cc:5495] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40f2b45e4571412f8b0c815ca46eaf90 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "40f2b45e4571412f8b0c815ca46eaf90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40f2b45e4571412f8b0c815ca46eaf90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 } health_report { overall_health: UNKNOWN } } }
I20260924 04:36:10.176518 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.173509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.0.98.1:32965)
0924 04:36:10.173610 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.173615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.173630 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:10.173775 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.173777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.173824 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.173905 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:10.174131 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.174596 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.175366 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:10.175563 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:10.175636 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:10.175874 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:10.176248 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:10.176334 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.176382 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.176423 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:36:10.182114 4894 raft_consensus.cc:1261] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 1 FOLLOWER]: Refusing update from remote peer 40f2b45e4571412f8b0c815ca46eaf90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:10.182206 4761 raft_consensus.cc:1261] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 1 FOLLOWER]: Refusing update from remote peer 40f2b45e4571412f8b0c815ca46eaf90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:10.182646 4963 consensus_queue.cc:1035] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07957d3f567240e3981938a01b06ab59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:10.182825 4970 consensus_queue.cc:1035] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "109cf9de82664b5cbc3b5fce06b251a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:10.190706 4975 mvcc.cc:205] Tried to move back new op lower bound from 7332759839459246080 to 7332759839384117248. Current Snapshot: MvccSnapshot[applied={T|T < 7332759839459246080}]
I20260924 04:36:10.199743 4483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:10.199882 4483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:10.202070 4483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b75cd7000d94e2eab8dc66219e553ac: Sending DeleteTablet for 3 replicas of tablet 51a3ef5da8084f2babde5afd7753f84c
I20260924 04:36:10.202737 4741 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a3ef5da8084f2babde5afd7753f84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54734
I20260924 04:36:10.202874 4874 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a3ef5da8084f2babde5afd7753f84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36888
I20260924 04:36:10.203023 4608 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a3ef5da8084f2babde5afd7753f84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47458
I20260924 04:36:10.203410 4983 tablet_replica.cc:330] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: stopping tablet replica
I20260924 04:36:10.203529 4985 tablet_replica.cc:330] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90: stopping tablet replica
I20260924 04:36:10.203799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 4546
I20260924 04:36:10.203809 4985 raft_consensus.cc:2229] T 51a3ef5da8084f2babde5afd7753f84c P 40f2b45e4571412f8b0c815ca46eaf90 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:10.203810 4983 raft_consensus.cc:2229] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:10.204641 4983 raft_consensus.cc:2258] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:10.206152 4983 ts_tablet_manager.cc:1905] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:10.209378 4984 tablet_replica.cc:330] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: stopping tablet replica
I20260924 04:36:10.209626 4983 ts_tablet_manager.cc:1918] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:10.209726 4983 log.cc:1196] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/51a3ef5da8084f2babde5afd7753f84c
I20260924 04:36:10.209712 4984 raft_consensus.cc:2229] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:10.209986 4984 raft_consensus.cc:2258] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:10.210398 4983 ts_tablet_manager.cc:1939] T 51a3ef5da8084f2babde5afd7753f84c P 109cf9de82664b5cbc3b5fce06b251a5: Deleting consensus metadata
I20260924 04:36:10.211274 4984 ts_tablet_manager.cc:1905] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:36:10.211707 4469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32965 (error 108)
I20260924 04:36:10.211886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 4679
I20260924 04:36:10.212164 4469 catalog_manager.cc:4841] TS 109cf9de82664b5cbc3b5fce06b251a5 (127.0.98.2:42621): tablet 51a3ef5da8084f2babde5afd7753f84c (table test-table [id=b5236fe15d4c4d7ea64b26cc67a10612]) successfully deleted
I20260924 04:36:10.213361 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.212986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32965 (local address 127.0.0.1:47474)
0924 04:36:10.213190 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.213262 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 04:36:10.213539 4469 catalog_manager.cc:4584] TS 40f2b45e4571412f8b0c815ca46eaf90 (127.0.98.1:32965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51a3ef5da8084f2babde5afd7753f84c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32965: connect: Connection refused (error 111)
I20260924 04:36:10.214395 4984 ts_tablet_manager.cc:1918] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:10.214493 4984 log.cc:1196] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/51a3ef5da8084f2babde5afd7753f84c
I20260924 04:36:10.214788 4984 ts_tablet_manager.cc:1939] T 51a3ef5da8084f2babde5afd7753f84c P 07957d3f567240e3981938a01b06ab59: Deleting consensus metadata
I20260924 04:36:10.215764 4469 catalog_manager.cc:4841] TS 07957d3f567240e3981938a01b06ab59 (127.0.98.3:32875): tablet 51a3ef5da8084f2babde5afd7753f84c (table test-table [id=b5236fe15d4c4d7ea64b26cc67a10612]) successfully deleted
I20260924 04:36:10.218279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 4812
I20260924 04:36:10.223904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 4453
2026-09-24T04:36:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2207 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T04:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:10Z Disabled control of system clock
I20260924 04:36:10.252822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40569
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:36:10.359021 4990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:10.359269 4990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:10.359306 4990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:10.363226 4990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:10.363276 4990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:10.363292 4990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:10.363309 4990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:10.363360 4990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:10.368085 4990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:10.369095 4990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:10.370442 4990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:10.378628 4996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:10.378702 4998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:10.378710 4995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:10.378804 4990 server_base.cc:1034] running on GCE node
I20260924 04:36:10.379388 4990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:10.380510 4990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:10.381690 4990 hybrid_clock.cc:648] HybridClock initialized: now 1790224570381661 us; error 49 us; skew 500 ppm
I20260924 04:36:10.395104 4990 webserver.cc:460] Webserver started at http://127.0.98.62:40281/ using document root <none> and password file <none>
I20260924 04:36:10.395820 4990 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:10.395913 4990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:10.396139 4990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:10.398267 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "ae000eb4cdaa4c45b2b11008f37bd36f"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "b11a724a44289b394f7f5185f9ccd6c0"
server_key_iv: "11d868f2b9739a97cbadaeee48887f84"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.398828 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae000eb4cdaa4c45b2b11008f37bd36f"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "b11a724a44289b394f7f5185f9ccd6c0"
server_key_iv: "11d868f2b9739a97cbadaeee48887f84"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.402313 4990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:10.404503 5004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.405678 4990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:10.405809 4990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "ae000eb4cdaa4c45b2b11008f37bd36f"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "b11a724a44289b394f7f5185f9ccd6c0"
server_key_iv: "11d868f2b9739a97cbadaeee48887f84"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.405910 4990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:10.410173 4990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:10.410727 4990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:10.410897 4990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:10.417083 4990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40569
I20260924 04:36:10.417094 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40569 every 8 connection(s)
I20260924 04:36:10.417971 4990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:10.420898 5057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:10.426095 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f: Bootstrap starting.
I20260924 04:36:10.427799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 4990
I20260924 04:36:10.427923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:10.428184 392 external_mini_cluster.cc:1428] Setting key 9b3058606e02b11365557bafd3e6fcea
I20260924 04:36:10.428468 5057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:10.429340 5057 log.cc:824] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:10.431107 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f: No bootstrap required, opened a new log
I20260924 04:36:10.433068 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.429301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.0.98.62:40569)
0924 04:36:10.429931 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.429946 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.429979 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:36:10.430557 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.430572 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.430837 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.431101 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:10.431896 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:10.431901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:10.431912 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:10.431921 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:10.431970 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:10.431985 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:10.432069 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.432198 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.432443 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":155,"threads_started":1}
I20260924 04:36:10.434983 5057 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae000eb4cdaa4c45b2b11008f37bd36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } }
I20260924 04:36:10.435240 5057 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:10.435300 5057 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae000eb4cdaa4c45b2b11008f37bd36f, State: Initialized, Role: FOLLOWER
I20260924 04:36:10.435704 5057 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae000eb4cdaa4c45b2b11008f37bd36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } }
I20260924 04:36:10.435837 5057 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:10.435899 5057 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:10.435992 5057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:10.436790 5057 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae000eb4cdaa4c45b2b11008f37bd36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } }
I20260924 04:36:10.437105 5057 leader_election.cc:302] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae000eb4cdaa4c45b2b11008f37bd36f; no voters:
I20260924 04:36:10.437425 5057 leader_election.cc:288] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:10.437795 5063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:10.438252 5063 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [term 1 LEADER]: Becoming Leader. State: Replica: ae000eb4cdaa4c45b2b11008f37bd36f, State: Running, Role: LEADER
I20260924 04:36:10.438526 5063 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae000eb4cdaa4c45b2b11008f37bd36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } }
I20260924 04:36:10.438637 5057 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:10.440171 5064 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae000eb4cdaa4c45b2b11008f37bd36f. Latest consensus state: current_term: 1 leader_uuid: "ae000eb4cdaa4c45b2b11008f37bd36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae000eb4cdaa4c45b2b11008f37bd36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } } }
I20260924 04:36:10.440294 5064 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:10.440515 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae000eb4cdaa4c45b2b11008f37bd36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae000eb4cdaa4c45b2b11008f37bd36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } } }
I20260924 04:36:10.440619 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f [sys.catalog]: This master's current role is: LEADER
W20260924 04:36:10.443490 5077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:36:10.443568 5077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:36:10.443621 5076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:10.444432 5076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:10.449815 5076 catalog_manager.cc:1347] Generated new cluster ID: 31d666f03a294b0d946f716754cfabce
I20260924 04:36:10.449878 5076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:10.487849 5076 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:10.488190 5076 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:10.495225 5076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f: Generated new TSK 0
I20260924 04:36:10.495993 5076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:36:10.499770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40569
--builtin_ntp_servers=127.0.98.20:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:36:10.611199 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:10.611464 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:10.611548 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:10.615509 5081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:10.615582 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:10.615669 5081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:10.620236 5081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:10.621326 5081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:10.622808 5081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:10.631906 5086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:10.631928 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:10.631934 5089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:10.632165 5081 server_base.cc:1034] running on GCE node
I20260924 04:36:10.632587 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:10.633246 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:10.634413 5081 hybrid_clock.cc:648] HybridClock initialized: now 1790224570634382 us; error 48 us; skew 500 ppm
I20260924 04:36:10.636149 5081 webserver.cc:460] Webserver started at http://127.0.98.1:39403/ using document root <none> and password file <none>
I20260924 04:36:10.636690 5081 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:10.636782 5081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:10.637002 5081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:10.639029 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "e2511c89d1964278861e6e68b16d2ae6"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "c38c31f01551949faf3b58a1a7a25170"
server_key_iv: "c582c9854118d1ffb8444a5489ef91ea"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.639638 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e2511c89d1964278861e6e68b16d2ae6"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "c38c31f01551949faf3b58a1a7a25170"
server_key_iv: "c582c9854118d1ffb8444a5489ef91ea"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.643227 5081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:10.645539 5095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.646683 5081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:10.646806 5081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "e2511c89d1964278861e6e68b16d2ae6"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "c38c31f01551949faf3b58a1a7a25170"
server_key_iv: "c582c9854118d1ffb8444a5489ef91ea"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.646908 5081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:10.651150 5081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:10.651679 5081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:10.651842 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:10.652338 5081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:10.653203 5081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:10.653270 5081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.653330 5081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:10.653376 5081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.661720 5081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39495
I20260924 04:36:10.661741 5208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39495 every 8 connection(s)
I20260924 04:36:10.662519 5081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:10.665901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 5081
I20260924 04:36:10.666009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:10.666291 392 external_mini_cluster.cc:1428] Setting key e9a61bda3f7bbeb58511728b8d887b5a
I20260924 04:36:10.668377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40569
--builtin_ntp_servers=127.0.98.20:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:36:10.668610 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.664162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37207 (local address 127.0.98.62:40569)
0924 04:36:10.664317 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.664321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.665327 (+ 1006us) server_negotiation.cc:407] Connection header received
0924 04:36:10.666747 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.666750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.666817 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.666909 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:10.668111 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:10.668118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:10.668124 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:10.668140 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:10.668166 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:10.668180 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:10.668281 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.668390 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.668434 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:36:10.669399 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.664508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40569 (local address 127.0.98.1:37207)
0924 04:36:10.665180 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.665230 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.666512 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.667109 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.667117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.667219 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:10.667869 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:10.667870 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:10.667906 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:10.667968 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:10.668266 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:10.668272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:10.668275 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.668396 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.668638 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260924 04:36:10.670554 5209 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40569
I20260924 04:36:10.670799 5209 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:10.671226 5209 heartbeater.cc:511] Master 127.0.98.62:40569 requested a full tablet report, sending...
I20260924 04:36:10.672643 5021 ts_manager.cc:194] Registered new tserver with Master: e2511c89d1964278861e6e68b16d2ae6 (127.0.98.1:39495)
W20260924 04:36:10.777042 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:10.777293 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:10.777370 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:10.781272 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:10.781314 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:10.781405 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:10.786599 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:10.787595 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:10.788913 5212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:10.797953 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:10.798077 5212 server_base.cc:1034] running on GCE node
W20260924 04:36:10.797989 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:10.797958 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:10.798593 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:10.799202 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:10.800431 5212 hybrid_clock.cc:648] HybridClock initialized: now 1790224570800394 us; error 49 us; skew 500 ppm
I20260924 04:36:10.802202 5212 webserver.cc:460] Webserver started at http://127.0.98.2:46599/ using document root <none> and password file <none>
I20260924 04:36:10.802745 5212 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:10.802829 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:10.803043 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:10.804960 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "89c81e21a17641248d925b59f742236f"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "7edd121214203cf4c04046d98f1321f4"
server_key_iv: "10c2b4e81e813e692bd3311e7899e2c0"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.805563 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "89c81e21a17641248d925b59f742236f"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "7edd121214203cf4c04046d98f1321f4"
server_key_iv: "10c2b4e81e813e692bd3311e7899e2c0"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.809011 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:36:10.811337 5226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.812407 5212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:10.812556 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "89c81e21a17641248d925b59f742236f"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "7edd121214203cf4c04046d98f1321f4"
server_key_iv: "10c2b4e81e813e692bd3311e7899e2c0"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.812659 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:10.816854 5212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:10.817412 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:10.817587 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:10.818082 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:10.818917 5212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:10.818984 5212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.819039 5212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:10.819084 5212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.827095 5212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42329
I20260924 04:36:10.827167 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42329 every 8 connection(s)
I20260924 04:36:10.827921 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:10.833577 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.829715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54681 (local address 127.0.98.62:40569)
0924 04:36:10.829855 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.829867 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.830788 (+ 921us) server_negotiation.cc:407] Connection header received
0924 04:36:10.832002 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:10.832005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:10.832043 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:10.832119 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:10.833122 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:10.833125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:10.833127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:10.833181 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:10.833208 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:10.833218 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:10.833295 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:10.833403 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 04:36:10.833440 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:36:10.834312 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.830069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40569 (local address 127.0.98.2:54681)
0924 04:36:10.830657 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.830695 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:10.831815 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:10.832267 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:10.832273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:10.832355 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:10.832874 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:10.832876 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:10.832912 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:10.832975 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:10.833283 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:10.833289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:10.833293 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:36:10.833422 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 04:36:10.833600 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":105,"threads_started":1}
I20260924 04:36:10.834306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 5212
I20260924 04:36:10.834457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:10.834813 392 external_mini_cluster.cc:1428] Setting key 54f738383e0a16deea6a6cf3a5390bde
I20260924 04:36:10.835373 5340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40569
I20260924 04:36:10.835584 5340 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:10.835974 5340 heartbeater.cc:511] Master 127.0.98.62:40569 requested a full tablet report, sending...
I20260924 04:36:10.836788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40569
--builtin_ntp_servers=127.0.98.20:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:36:10.836912 5021 ts_manager.cc:194] Registered new tserver with Master: 89c81e21a17641248d925b59f742236f (127.0.98.2:42329)
W20260924 04:36:10.944115 5343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:10.944370 5343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:10.944442 5343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:10.948499 5343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:10.948572 5343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:10.948652 5343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:10.953428 5343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:10.954391 5343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:10.955693 5343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:10.964746 5349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:10.964892 5343 server_base.cc:1034] running on GCE node
W20260924 04:36:10.964807 5351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:10.964749 5348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:10.965469 5343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:10.966097 5343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:10.967276 5343 hybrid_clock.cc:648] HybridClock initialized: now 1790224570967258 us; error 35 us; skew 500 ppm
I20260924 04:36:10.969069 5343 webserver.cc:460] Webserver started at http://127.0.98.3:41983/ using document root <none> and password file <none>
I20260924 04:36:10.969662 5343 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:10.969749 5343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:10.969950 5343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:10.971946 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ba90e2324a841319bb64fc4b88cb68d"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "872676e4e7b05ebbddfbef5877708123"
server_key_iv: "1ea97a2d3e09809fcd3c2c3c0e39061e"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.972506 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ba90e2324a841319bb64fc4b88cb68d"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "872676e4e7b05ebbddfbef5877708123"
server_key_iv: "1ea97a2d3e09809fcd3c2c3c0e39061e"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.976105 5343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:36:10.978487 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.979709 5343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:10.979843 5343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "5ba90e2324a841319bb64fc4b88cb68d"
format_stamp: "Formatted at 2026-09-24 04:36:10 on dist-test-slave-jlnt"
server_key: "872676e4e7b05ebbddfbef5877708123"
server_key_iv: "1ea97a2d3e09809fcd3c2c3c0e39061e"
server_key_version: "encryptionkey@0"
I20260924 04:36:10.979951 5343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:10.984648 5343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:10.985275 5343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:10.985441 5343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:10.985939 5343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:10.986784 5343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:10.986850 5343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.986910 5343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:10.986953 5343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:10.995666 5343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38343
I20260924 04:36:10.995692 5470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38343 every 8 connection(s)
I20260924 04:36:10.996505 5343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:11.002048 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.998235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36533 (local address 127.0.98.62:40569)
0924 04:36:10.998384 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:10.998388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:10.999345 (+ 957us) server_negotiation.cc:407] Connection header received
0924 04:36:11.000559 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.000561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.000596 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.000653 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.001695 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.001698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.001700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.001712 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.001730 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.001738 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.001819 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.001910 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.001943 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:36:11.002494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 5343
I20260924 04:36:11.002604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:11.002849 392 external_mini_cluster.cc:1428] Setting key ad0c5ccecd9a7491f7d1c5725d5aab09
I20260924 04:36:11.002969 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:10.998619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40569 (local address 127.0.98.3:36533)
0924 04:36:10.999215 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:10.999263 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:11.000384 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:11.000784 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:11.000790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:11.000873 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:11.001421 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:11.001423 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:11.001460 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:11.001526 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:11.001808 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:11.001812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:11.001816 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:36:11.001930 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:36:11.002127 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":109,"threads_started":1}
I20260924 04:36:11.003975 5471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40569
I20260924 04:36:11.004201 5471 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:11.004611 5471 heartbeater.cc:511] Master 127.0.98.62:40569 requested a full tablet report, sending...
I20260924 04:36:11.005575 5020 ts_manager.cc:194] Registered new tserver with Master: 5ba90e2324a841319bb64fc4b88cb68d (127.0.98.3:38343)
I20260924 04:36:11.016374 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:36:11.019842 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.018177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.0.98.62:40569)
0924 04:36:11.018312 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.018317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.018496 (+ 179us) server_negotiation.cc:407] Connection header received
0924 04:36:11.018669 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.018673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.018708 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.018787 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.019449 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.019453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.019456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.019467 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.019490 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.019498 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.019581 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.019673 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.019712 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:36:11.022413 5020 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:36:11.025341 5020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:11.036592 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.033944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.0.98.1:39495)
0924 04:36:11.034259 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.034272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.034419 (+ 147us) server_negotiation.cc:407] Connection header received
0924 04:36:11.034487 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.034493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.034705 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.035212 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.036044 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.036050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.036062 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.036072 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.036114 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.036142 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.036266 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.036371 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.036436 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
I20260924 04:36:11.036557 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.033377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39495 (local address 127.0.0.1:37756)
0924 04:36:11.034315 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:11.034329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:11.034406 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:11.034877 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:11.034886 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:11.034958 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:11.035506 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:11.035509 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:11.035531 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:11.035609 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:11.036238 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:11.036247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:11.036252 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:36:11.036360 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:36:11.036425 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":79,"threads_started":1}
I20260924 04:36:11.037173 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.033978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38343 (local address 127.0.0.1:43152)
0924 04:36:11.034677 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:11.034690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:11.034753 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:11.035490 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:11.035493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:11.035504 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:11.036373 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:11.036375 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:11.036385 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:11.036408 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:11.036921 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:11.036924 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:11.036927 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:36:11.036991 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:36:11.037020 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":111,"threads_started":1}
I20260924 04:36:11.037201 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.033306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.0.98.2:42329)
0924 04:36:11.033909 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.033922 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.033946 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:36:11.034519 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.034526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.034724 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.034836 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.036124 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.036130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.036142 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.036153 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.036194 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.036218 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.036318 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.036938 (+ 620us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.037008 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":99,"threads_started":1}
I20260924 04:36:11.037621 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.034410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.0.98.3:38343)
0924 04:36:11.034831 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.034844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.034865 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:36:11.034910 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.034914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.035062 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.035211 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.036543 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.036561 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.036571 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.036582 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.036624 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.036779 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.036882 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.037383 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.037450 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":143,"threads_started":1}
I20260924 04:36:11.037479 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.033191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42329 (local address 127.0.0.1:44100)
0924 04:36:11.033516 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:11.033564 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:11.034380 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:11.035454 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:11.035462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:11.035479 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:11.035962 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:11.035966 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:11.035979 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:11.036009 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:11.036305 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:11.036309 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:11.036312 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:36:11.037339 (+ 1027us) client_negotiation.cc:240] Negotiation successful
0924 04:36:11.037375 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260924 04:36:11.039301 5143 tablet_service.cc:1467] Processing CreateTablet for tablet babb4c972b3a4d8aa98c7e849c02c3b9 (DEFAULT_TABLE table=test-table [id=f1abe46cd68d4de6b3e44f92a854b4aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:11.040270 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet babb4c972b3a4d8aa98c7e849c02c3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:11.040089 5405 tablet_service.cc:1467] Processing CreateTablet for tablet babb4c972b3a4d8aa98c7e849c02c3b9 (DEFAULT_TABLE table=test-table [id=f1abe46cd68d4de6b3e44f92a854b4aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:11.040889 5405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet babb4c972b3a4d8aa98c7e849c02c3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:11.046277 5488 tablet_bootstrap.cc:492] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6: Bootstrap starting.
I20260924 04:36:11.047096 5274 tablet_service.cc:1467] Processing CreateTablet for tablet babb4c972b3a4d8aa98c7e849c02c3b9 (DEFAULT_TABLE table=test-table [id=f1abe46cd68d4de6b3e44f92a854b4aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:11.047529 5489 tablet_bootstrap.cc:492] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: Bootstrap starting.
I20260924 04:36:11.047959 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet babb4c972b3a4d8aa98c7e849c02c3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:11.048563 5488 tablet_bootstrap.cc:654] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:11.049445 5488 log.cc:824] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:11.049636 5489 tablet_bootstrap.cc:654] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:11.050439 5489 log.cc:824] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:11.051537 5488 tablet_bootstrap.cc:492] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6: No bootstrap required, opened a new log
I20260924 04:36:11.051741 5488 ts_tablet_manager.cc:1397] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:36:11.052084 5489 tablet_bootstrap.cc:492] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: No bootstrap required, opened a new log
I20260924 04:36:11.052258 5489 ts_tablet_manager.cc:1397] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:36:11.054641 5492 tablet_bootstrap.cc:492] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: Bootstrap starting.
I20260924 04:36:11.055953 5488 raft_consensus.cc:348] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } }
I20260924 04:36:11.056180 5488 raft_consensus.cc:374] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:11.056239 5488 raft_consensus.cc:729] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2511c89d1964278861e6e68b16d2ae6, State: Initialized, Role: FOLLOWER
I20260924 04:36:11.056167 5489 raft_consensus.cc:348] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } }
I20260924 04:36:11.056407 5489 raft_consensus.cc:374] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:11.056514 5489 raft_consensus.cc:729] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba90e2324a841319bb64fc4b88cb68d, State: Initialized, Role: FOLLOWER
I20260924 04:36:11.056702 5488 consensus_queue.cc:260] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } }
I20260924 04:36:11.056972 5489 consensus_queue.cc:260] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } }
I20260924 04:36:11.057096 5492 tablet_bootstrap.cc:654] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:11.057658 5471 heartbeater.cc:503] Master 127.0.98.62:40569 was elected leader, sending a full tablet report...
I20260924 04:36:11.057708 5488 ts_tablet_manager.cc:1428] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:36:11.057998 5209 heartbeater.cc:503] Master 127.0.98.62:40569 was elected leader, sending a full tablet report...
I20260924 04:36:11.058177 5492 log.cc:824] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:11.058697 5489 ts_tablet_manager.cc:1428] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:36:11.060654 5492 tablet_bootstrap.cc:492] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: No bootstrap required, opened a new log
I20260924 04:36:11.060830 5492 ts_tablet_manager.cc:1397] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260924 04:36:11.064409 5492 raft_consensus.cc:348] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } }
I20260924 04:36:11.064626 5492 raft_consensus.cc:374] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:11.064697 5492 raft_consensus.cc:729] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c81e21a17641248d925b59f742236f, State: Initialized, Role: FOLLOWER
I20260924 04:36:11.065093 5492 consensus_queue.cc:260] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } }
I20260924 04:36:11.065687 5340 heartbeater.cc:503] Master 127.0.98.62:40569 was elected leader, sending a full tablet report...
I20260924 04:36:11.066002 5492 ts_tablet_manager.cc:1428] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:36:11.072301 5496 raft_consensus.cc:482] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:11.072454 5496 raft_consensus.cc:504] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } }
I20260924 04:36:11.073446 5496 leader_election.cc:288] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2511c89d1964278861e6e68b16d2ae6 (127.0.98.1:39495), 5ba90e2324a841319bb64fc4b88cb68d (127.0.98.3:38343)
I20260924 04:36:11.075264 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.073640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39495 (local address 127.0.98.2:50699)
0924 04:36:11.073821 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:11.073838 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:11.073911 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:11.074180 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:11.074185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:11.074201 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:11.074736 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:11.074737 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:11.074746 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:11.074771 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:11.075009 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:11.075015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:11.075020 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:36:11.075105 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:36:11.075133 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 04:36:11.075309 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.073742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50699 (local address 127.0.98.1:39495)
0924 04:36:11.073868 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.073873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.073886 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:11.074017 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.074021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.074059 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.074155 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.074884 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.074887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.074889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.074900 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.074917 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.074926 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.074997 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.075127 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.075170 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:36:11.075479 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.073783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38993 (local address 127.0.98.3:38343)
0924 04:36:11.073898 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.073903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.074195 (+ 292us) server_negotiation.cc:407] Connection header received
0924 04:36:11.074275 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.074280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.074321 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.074406 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.075071 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.075090 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.075099 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.075112 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.075134 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.075143 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.075225 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.075315 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.075355 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:36:11.075603 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.073650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38343 (local address 127.0.98.2:38993)
0924 04:36:11.074098 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:11.074114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:11.074196 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:11.074436 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:11.074440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:11.074454 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:11.074927 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:11.074928 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:11.074936 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:11.074955 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:11.075225 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:11.075228 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:11.075231 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:36:11.075318 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:36:11.075472 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":102,"thread_start_us":95,"threads_started":1}
I20260924 04:36:11.075834 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babb4c972b3a4d8aa98c7e849c02c3b9" candidate_uuid: "89c81e21a17641248d925b59f742236f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2511c89d1964278861e6e68b16d2ae6" is_pre_election: true
I20260924 04:36:11.076099 5163 raft_consensus.cc:2454] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c81e21a17641248d925b59f742236f in term 0.
I20260924 04:36:11.076184 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babb4c972b3a4d8aa98c7e849c02c3b9" candidate_uuid: "89c81e21a17641248d925b59f742236f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba90e2324a841319bb64fc4b88cb68d" is_pre_election: true
I20260924 04:36:11.076445 5425 raft_consensus.cc:2454] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c81e21a17641248d925b59f742236f in term 0.
I20260924 04:36:11.076560 5229 leader_election.cc:302] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89c81e21a17641248d925b59f742236f, e2511c89d1964278861e6e68b16d2ae6; no voters:
I20260924 04:36:11.076864 5496 raft_consensus.cc:2781] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:11.076951 5496 raft_consensus.cc:482] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:11.076996 5496 raft_consensus.cc:3037] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:11.078022 5496 raft_consensus.cc:504] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } }
I20260924 04:36:11.078313 5496 leader_election.cc:288] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [CANDIDATE]: Term 1 election: Requested vote from peers e2511c89d1964278861e6e68b16d2ae6 (127.0.98.1:39495), 5ba90e2324a841319bb64fc4b88cb68d (127.0.98.3:38343)
I20260924 04:36:11.078608 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babb4c972b3a4d8aa98c7e849c02c3b9" candidate_uuid: "89c81e21a17641248d925b59f742236f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2511c89d1964278861e6e68b16d2ae6"
I20260924 04:36:11.078657 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babb4c972b3a4d8aa98c7e849c02c3b9" candidate_uuid: "89c81e21a17641248d925b59f742236f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba90e2324a841319bb64fc4b88cb68d"
I20260924 04:36:11.078728 5163 raft_consensus.cc:3037] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:11.078753 5425 raft_consensus.cc:3037] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [term 0 FOLLOWER]: Advancing to term 1
W20260924 04:36:11.078821 5341 tablet.cc:2369] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:36:11.079962 5163 raft_consensus.cc:2454] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c81e21a17641248d925b59f742236f in term 1.
I20260924 04:36:11.079962 5425 raft_consensus.cc:2454] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c81e21a17641248d925b59f742236f in term 1.
I20260924 04:36:11.080272 5228 leader_election.cc:302] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba90e2324a841319bb64fc4b88cb68d, 89c81e21a17641248d925b59f742236f; no voters:
I20260924 04:36:11.080458 5496 raft_consensus.cc:2781] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:11.080677 5496 raft_consensus.cc:686] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 1 LEADER]: Becoming Leader. State: Replica: 89c81e21a17641248d925b59f742236f, State: Running, Role: LEADER
I20260924 04:36:11.080965 5496 consensus_queue.cc:237] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } }
I20260924 04:36:11.083976 5021 catalog_manager.cc:5495] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f reported cstate change: term changed from 0 to 1, leader changed from <none> to 89c81e21a17641248d925b59f742236f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "89c81e21a17641248d925b59f742236f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c81e21a17641248d925b59f742236f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 } health_report { overall_health: UNKNOWN } } }
I20260924 04:36:11.095762 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.094191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.0.98.2:42329)
0924 04:36:11.094415 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.094422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.094437 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:11.094615 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.094617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.094670 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.094754 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.095352 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.095356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.095358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.095369 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.095388 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.095401 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.095483 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.095593 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.095631 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260924 04:36:11.101004 5163 raft_consensus.cc:1261] T babb4c972b3a4d8aa98c7e849c02c3b9 P e2511c89d1964278861e6e68b16d2ae6 [term 1 FOLLOWER]: Refusing update from remote peer 89c81e21a17641248d925b59f742236f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:11.101004 5425 raft_consensus.cc:1261] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [term 1 FOLLOWER]: Refusing update from remote peer 89c81e21a17641248d925b59f742236f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:11.101723 5498 consensus_queue.cc:1035] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2511c89d1964278861e6e68b16d2ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:11.101857 5496 consensus_queue.cc:1035] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba90e2324a841319bb64fc4b88cb68d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:11.107929 5501 mvcc.cc:205] Tried to move back new op lower bound from 7332759843223359488 to 7332759843149950976. Current Snapshot: MvccSnapshot[applied={T|T < 7332759843223359488}]
I20260924 04:36:11.111958 5503 mvcc.cc:205] Tried to move back new op lower bound from 7332759843223359488 to 7332759843149950976. Current Snapshot: MvccSnapshot[applied={T|T < 7332759843223359488}]
I20260924 04:36:11.115814 5020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:11.115972 5020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:11.118378 5020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae000eb4cdaa4c45b2b11008f37bd36f: Sending DeleteTablet for 3 replicas of tablet babb4c972b3a4d8aa98c7e849c02c3b9
I20260924 04:36:11.119078 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet babb4c972b3a4d8aa98c7e849c02c3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:11 UTC) from {username='slave'} at 127.0.0.1:44100
I20260924 04:36:11.119072 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet babb4c972b3a4d8aa98c7e849c02c3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:11 UTC) from {username='slave'} at 127.0.0.1:37756
I20260924 04:36:11.119252 5405 tablet_service.cc:1514] Processing DeleteTablet for tablet babb4c972b3a4d8aa98c7e849c02c3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:11 UTC) from {username='slave'} at 127.0.0.1:43152
I20260924 04:36:11.119637 5512 tablet_replica.cc:330] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: stopping tablet replica
I20260924 04:36:11.119905 5512 raft_consensus.cc:2229] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:11.120168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 5081
I20260924 04:36:11.122421 5512 raft_consensus.cc:2258] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:11.123592 5512 ts_tablet_manager.cc:1905] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:11.125810 5512 ts_tablet_manager.cc:1918] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:11.125911 5512 log.cc:1196] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/babb4c972b3a4d8aa98c7e849c02c3b9
I20260924 04:36:11.126199 5512 ts_tablet_manager.cc:1939] T babb4c972b3a4d8aa98c7e849c02c3b9 P 89c81e21a17641248d925b59f742236f: Deleting consensus metadata
I20260924 04:36:11.127180 5005 catalog_manager.cc:4841] TS 89c81e21a17641248d925b59f742236f (127.0.98.2:42329): tablet babb4c972b3a4d8aa98c7e849c02c3b9 (table test-table [id=f1abe46cd68d4de6b3e44f92a854b4aa]) successfully deleted
I20260924 04:36:11.129344 5513 tablet_replica.cc:330] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: stopping tablet replica
I20260924 04:36:11.129647 5513 raft_consensus.cc:2229] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:11.129921 5513 raft_consensus.cc:2258] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:36:11.130331 5007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39495 (error 108)
I20260924 04:36:11.130486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 5212
I20260924 04:36:11.132568 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.132238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39495 (local address 127.0.0.1:37762)
0924 04:36:11.132371 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:11.132454 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 04:36:11.132766 5007 catalog_manager.cc:4584] TS e2511c89d1964278861e6e68b16d2ae6 (127.0.98.1:39495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet babb4c972b3a4d8aa98c7e849c02c3b9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39495: connect: Connection refused (error 111)
I20260924 04:36:11.133358 5513 ts_tablet_manager.cc:1905] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:11.135361 5513 ts_tablet_manager.cc:1918] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:11.135434 5513 log.cc:1196] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/babb4c972b3a4d8aa98c7e849c02c3b9
I20260924 04:36:11.135699 5513 ts_tablet_manager.cc:1939] T babb4c972b3a4d8aa98c7e849c02c3b9 P 5ba90e2324a841319bb64fc4b88cb68d: Deleting consensus metadata
I20260924 04:36:11.136433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 5343
I20260924 04:36:11.142107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 4990
2026-09-24T04:36:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (917 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 04:36:11.162766 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:36:11.164316 392 webserver.cc:460] Webserver started at https://127.0.0.1:37405/ using document root <none> and password file <none>
I20260924 04:36:11.164492 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:36:11.164758 392 webserver.cc:460] Webserver started at http://127.0.0.1:45493/ using document root <none> and password file <none>
2026-09-24T04:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:11Z Disabled control of system clock
I20260924 04:36:11.173492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41621
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41621
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37405/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/testchainca.cert with env {}
W20260924 04:36:11.280913 5520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:11.281229 5520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:11.281337 5520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:11.285343 5520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:11.285423 5520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:11.285451 5520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:11.285490 5520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:11.285526 5520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:36:11.285559 5520 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:36:11.285589 5520 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37405/jwks.json
I20260924 04:36:11.290814 5520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37405/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:11.291824 5520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:11.292801 5520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:11.300942 5525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:11.300930 5526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:11.301011 5520 server_base.cc:1034] running on GCE node
W20260924 04:36:11.300946 5528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:11.301689 5520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:11.302703 5520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:11.303874 5520 hybrid_clock.cc:648] HybridClock initialized: now 1790224571303847 us; error 40 us; skew 500 ppm
I20260924 04:36:11.305665 5520 webserver.cc:460] Webserver started at http://127.0.98.62:35567/ using document root <none> and password file <none>
I20260924 04:36:11.306211 5520 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:11.306298 5520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:11.306506 5520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:11.308419 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "ef2854314e0c4a5f819afba89c630bdc"
format_stamp: "Formatted at 2026-09-24 04:36:11 on dist-test-slave-jlnt"
server_key: "22cabaf5f00724468a1e84032315a083"
server_key_iv: "ecc88a4b1a3684031abf8c8cdcea2b80"
server_key_version: "encryptionkey@0"
I20260924 04:36:11.308969 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef2854314e0c4a5f819afba89c630bdc"
format_stamp: "Formatted at 2026-09-24 04:36:11 on dist-test-slave-jlnt"
server_key: "22cabaf5f00724468a1e84032315a083"
server_key_iv: "ecc88a4b1a3684031abf8c8cdcea2b80"
server_key_version: "encryptionkey@0"
I20260924 04:36:11.312372 5520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:36:11.314694 5534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:11.315796 5520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:11.315923 5520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "ef2854314e0c4a5f819afba89c630bdc"
format_stamp: "Formatted at 2026-09-24 04:36:11 on dist-test-slave-jlnt"
server_key: "22cabaf5f00724468a1e84032315a083"
server_key_iv: "ecc88a4b1a3684031abf8c8cdcea2b80"
server_key_version: "encryptionkey@0"
I20260924 04:36:11.316023 5520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:11.337805 5520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:11.338487 5520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:11.338694 5520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:11.345613 5586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41621 every 8 connection(s)
I20260924 04:36:11.345628 5520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41621
I20260924 04:36:11.346565 5520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:11.349295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 5520
I20260924 04:36:11.349383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:11.349681 392 external_mini_cluster.cc:1428] Setting key 08e090dfda2d0e6ca034ae29093f8aa9
I20260924 04:36:11.349804 5587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:11.355751 5587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc: Bootstrap starting.
I20260924 04:36:11.358086 5587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:11.358510 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.351033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.0.98.62:41621)
0924 04:36:11.351649 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.351663 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.351697 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:36:11.352318 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.352334 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.352716 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.353044 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.353494 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.354421 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.355330 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.355793 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.355961 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:11.356798 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.356803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.356814 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.357233 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.357288 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.357308 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.357408 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.357549 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.357797 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":133,"threads_started":1}
I20260924 04:36:11.358832 5587 log.cc:824] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:11.360751 5587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc: No bootstrap required, opened a new log
I20260924 04:36:11.361039 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:36:11.364261 5587 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2854314e0c4a5f819afba89c630bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } }
I20260924 04:36:11.364457 5587 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:11.364540 5587 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2854314e0c4a5f819afba89c630bdc, State: Initialized, Role: FOLLOWER
I20260924 04:36:11.364920 5587 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2854314e0c4a5f819afba89c630bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } }
I20260924 04:36:11.365032 5587 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:11.365075 5587 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:11.365195 5587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:11.366044 5587 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2854314e0c4a5f819afba89c630bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } }
I20260924 04:36:11.366326 5587 leader_election.cc:302] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef2854314e0c4a5f819afba89c630bdc; no voters:
I20260924 04:36:11.366551 5587 leader_election.cc:288] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:11.366658 5592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:11.366859 5592 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [term 1 LEADER]: Becoming Leader. State: Replica: ef2854314e0c4a5f819afba89c630bdc, State: Running, Role: LEADER
I20260924 04:36:11.367151 5592 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2854314e0c4a5f819afba89c630bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } }
I20260924 04:36:11.367662 5587 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:11.368707 5594 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2854314e0c4a5f819afba89c630bdc. Latest consensus state: current_term: 1 leader_uuid: "ef2854314e0c4a5f819afba89c630bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2854314e0c4a5f819afba89c630bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } }
I20260924 04:36:11.368700 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef2854314e0c4a5f819afba89c630bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2854314e0c4a5f819afba89c630bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } }
I20260924 04:36:11.368839 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:11.368839 5594 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:11.369230 5601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:11.372140 5601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:11.377583 5601 catalog_manager.cc:1347] Generated new cluster ID: 08c188338b0d47e49e332d74cf9da994
I20260924 04:36:11.377642 5601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:11.393275 5601 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:11.394371 5601 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:11.401860 5601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef2854314e0c4a5f819afba89c630bdc: Generated new TSK 0
I20260924 04:36:11.402532 5601 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:36:11.435161 5617 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 04:36:11.435278 5617 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:36:11.435763 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.411901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.0.98.62:41621)
0924 04:36:11.412018 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.412024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.412257 (+ 233us) server_negotiation.cc:407] Connection header received
0924 04:36:11.412488 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.412491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.412559 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.412665 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:36:11.412917 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.413496 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.414374 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.414591 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.414697 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:11.415021 (+ 324us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:36:11.435309 (+ 20288us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:36:11.435624 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36550: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 04:36:11.435838 5590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36550: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 04:36:11.437067 5616 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:11.411820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.0.1:36550)
0924 04:36:11.412174 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:11.412186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:11.412329 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:11.412684 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:11.412688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:11.412695 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:36:11.412813 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:11.412820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.413613 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.413617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:11.414255 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:11.414262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.414687 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.414712 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:11.414797 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:11.414845 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:36:11.435577 (+ 20732us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:36:11.435831 (+ 254us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41621: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41621: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 04:36:11.437875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 5520
2026-09-24T04:36:11Z chronyd exiting
I20260924 04:36:11.475244 392 test_util.cc:175] -----------------------------------------------
I20260924 04:36:11.475334 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790224545298817-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (313 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 04:36:11.476313 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:36:11.477182 392 webserver.cc:460] Webserver started at https://127.0.0.1:35187/ using document root <none> and password file <none>
I20260924 04:36:11.477283 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:36:11.477536 392 webserver.cc:460] Webserver started at http://127.0.0.1:42669/ using document root <none> and password file <none>
2026-09-24T04:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:11Z Disabled control of system clock
I20260924 04:36:11.485667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46617
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46617
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35187/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/testchainca.cert with env {}
W20260924 04:36:11.595925 5623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:11.596189 5623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:11.596262 5623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:11.600332 5623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:11.600409 5623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:11.600436 5623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:11.600472 5623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:11.600507 5623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:36:11.600541 5623 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:36:11.600572 5623 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35187/jwks.json
I20260924 04:36:11.605789 5623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35187/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:11.606807 5623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:11.607788 5623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:11.615976 5629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:11.616068 5623 server_base.cc:1034] running on GCE node
W20260924 04:36:11.616010 5628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:11.616014 5631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:11.616730 5623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:11.617810 5623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:11.618989 5623 hybrid_clock.cc:648] HybridClock initialized: now 1790224571618965 us; error 35 us; skew 500 ppm
I20260924 04:36:11.620800 5623 webserver.cc:460] Webserver started at http://127.0.98.62:43061/ using document root <none> and password file <none>
I20260924 04:36:11.621412 5623 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:11.621495 5623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:11.621707 5623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:11.623674 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "ea4588bcbb3c44d681db8a028a1635be"
format_stamp: "Formatted at 2026-09-24 04:36:11 on dist-test-slave-jlnt"
server_key: "cdb4d12c37cb646e43f51d5c775da82e"
server_key_iv: "81b08fd6d0f47304fd3b70af7b8aa185"
server_key_version: "encryptionkey@0"
I20260924 04:36:11.624214 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea4588bcbb3c44d681db8a028a1635be"
format_stamp: "Formatted at 2026-09-24 04:36:11 on dist-test-slave-jlnt"
server_key: "cdb4d12c37cb646e43f51d5c775da82e"
server_key_iv: "81b08fd6d0f47304fd3b70af7b8aa185"
server_key_version: "encryptionkey@0"
I20260924 04:36:11.627781 5623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:36:11.630215 5637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:11.631400 5623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:11.631547 5623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "ea4588bcbb3c44d681db8a028a1635be"
format_stamp: "Formatted at 2026-09-24 04:36:11 on dist-test-slave-jlnt"
server_key: "cdb4d12c37cb646e43f51d5c775da82e"
server_key_iv: "81b08fd6d0f47304fd3b70af7b8aa185"
server_key_version: "encryptionkey@0"
I20260924 04:36:11.631671 5623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:11.642130 5623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:11.642747 5623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:11.642925 5623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:11.649923 5623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46617
I20260924 04:36:11.649964 5689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46617 every 8 connection(s)
I20260924 04:36:11.650880 5623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:11.653889 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:11.659260 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be: Bootstrap starting.
I20260924 04:36:11.660876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 5623
I20260924 04:36:11.660983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:11.661305 392 external_mini_cluster.cc:1428] Setting key e79efb061de14e4469df37765d778204
I20260924 04:36:11.661935 5690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:11.662829 5690 log.cc:824] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:11.664647 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be: No bootstrap required, opened a new log
I20260924 04:36:11.668012 5690 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4588bcbb3c44d681db8a028a1635be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46617 } }
I20260924 04:36:11.668254 5690 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:11.668326 5690 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4588bcbb3c44d681db8a028a1635be, State: Initialized, Role: FOLLOWER
I20260924 04:36:11.668774 5690 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4588bcbb3c44d681db8a028a1635be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46617 } }
I20260924 04:36:11.668923 5690 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:11.668994 5690 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:11.669113 5690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:11.670035 5690 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4588bcbb3c44d681db8a028a1635be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46617 } }
I20260924 04:36:11.670085 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.662404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.98.62:46617)
0924 04:36:11.663079 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.663095 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.663129 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:36:11.663767 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.663784 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.664104 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.664431 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.664892 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.665917 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.666797 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.667260 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.667427 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:11.668275 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.668282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.668301 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.668751 (+ 450us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.668806 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.668827 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.668928 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.669064 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.669362 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":133,"threads_started":1}
I20260924 04:36:11.670439 5690 leader_election.cc:302] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea4588bcbb3c44d681db8a028a1635be; no voters:
I20260924 04:36:11.670756 5690 leader_election.cc:288] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:11.670857 5695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:11.671090 5695 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [term 1 LEADER]: Becoming Leader. State: Replica: ea4588bcbb3c44d681db8a028a1635be, State: Running, Role: LEADER
I20260924 04:36:11.671406 5695 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4588bcbb3c44d681db8a028a1635be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46617 } }
I20260924 04:36:11.671976 5690 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:11.672576 5696 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4588bcbb3c44d681db8a028a1635be. Latest consensus state: current_term: 1 leader_uuid: "ea4588bcbb3c44d681db8a028a1635be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4588bcbb3c44d681db8a028a1635be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46617 } } }
I20260924 04:36:11.672725 5696 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:11.672928 5697 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea4588bcbb3c44d681db8a028a1635be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4588bcbb3c44d681db8a028a1635be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46617 } } }
I20260924 04:36:11.673025 5697 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:11.673341 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:36:11.674307 5704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:11.677412 5704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:11.683135 5704 catalog_manager.cc:1347] Generated new cluster ID: d4fdde2b94e149368b2c1df94336a735
I20260924 04:36:11.683211 5704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:11.691061 5704 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:11.691983 5704 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:11.698647 5704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea4588bcbb3c44d681db8a028a1635be: Generated new TSK 0
I20260924 04:36:11.699349 5704 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:36:11.724151 5720 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 04:36:11.724244 5720 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:36:11.724907 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.709026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.0.98.62:46617)
0924 04:36:11.709180 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.709185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.709339 (+ 154us) server_negotiation.cc:407] Connection header received
0924 04:36:11.709494 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.709498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.709576 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.709707 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:36:11.709944 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.710778 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.711683 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.712033 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.712140 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:11.712432 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:36:11.724512 (+ 12080us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:36:11.724771 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45690: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:36:11.724977 5693 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45690: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 04:36:11.725008 5719 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:11.708895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46617 (local address 127.0.0.1:45690)
0924 04:36:11.709251 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:11.709263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:11.709360 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:11.709725 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:11.709728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:11.709735 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:36:11.709834 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:11.709840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.710922 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.710924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:11.711559 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:11.711566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.712150 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.712172 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:11.712242 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:11.712264 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:36:11.724704 (+ 12440us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:36:11.724840 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46617: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":137,"threads_started":1}
I20260924 04:36:11.725761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 5623
2026-09-24T04:36:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (288 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 04:36:11.764739 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:36:11.765681 392 webserver.cc:460] Webserver started at https://127.0.0.1:43977/ using document root <none> and password file <none>
I20260924 04:36:11.765799 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:36:11.766062 392 webserver.cc:460] Webserver started at http://127.0.0.1:45599/ using document root <none> and password file <none>
2026-09-24T04:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:11Z Disabled control of system clock
I20260924 04:36:11.774717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41289
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41289
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43977/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:36:11.881270 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:11.881517 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:11.881563 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:11.885404 5726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:11.885453 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:11.885468 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:11.885485 5726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:11.885499 5726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:36:11.885514 5726 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:36:11.885532 5726 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43977/jwks.json
I20260924 04:36:11.890091 5726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43977/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:11.891034 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:11.892314 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:11.900278 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:11.900314 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:11.900290 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:11.900576 5726 server_base.cc:1034] running on GCE node
I20260924 04:36:11.901253 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:11.902310 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:11.903506 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790224571903494 us; error 30 us; skew 500 ppm
I20260924 04:36:11.905267 5726 webserver.cc:460] Webserver started at http://127.0.98.62:41071/ using document root <none> and password file <none>
I20260924 04:36:11.905803 5726 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:11.905886 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:11.906090 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:11.908003 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "623084e1d3714df4b8e6415fb9ca6066"
format_stamp: "Formatted at 2026-09-24 04:36:11 on dist-test-slave-jlnt"
server_key: "eb2c1a3e954553f0cd0e3d6e631e3265"
server_key_iv: "435bb67c65b37fc20e59ad454e2d9301"
server_key_version: "encryptionkey@0"
I20260924 04:36:11.908557 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "623084e1d3714df4b8e6415fb9ca6066"
format_stamp: "Formatted at 2026-09-24 04:36:11 on dist-test-slave-jlnt"
server_key: "eb2c1a3e954553f0cd0e3d6e631e3265"
server_key_iv: "435bb67c65b37fc20e59ad454e2d9301"
server_key_version: "encryptionkey@0"
I20260924 04:36:11.912087 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 04:36:11.914484 5740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:11.915670 5726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:11.915805 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "623084e1d3714df4b8e6415fb9ca6066"
format_stamp: "Formatted at 2026-09-24 04:36:11 on dist-test-slave-jlnt"
server_key: "eb2c1a3e954553f0cd0e3d6e631e3265"
server_key_iv: "435bb67c65b37fc20e59ad454e2d9301"
server_key_version: "encryptionkey@0"
I20260924 04:36:11.915910 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:11.950424 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:11.951133 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:11.951329 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:11.957969 5726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41289
I20260924 04:36:11.958001 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41289 every 8 connection(s)
I20260924 04:36:11.958868 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:11.959841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 5726
I20260924 04:36:11.959964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:11.960256 392 external_mini_cluster.cc:1428] Setting key c1063014bf6f79dae72417444934184f
I20260924 04:36:11.962472 5793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:11.967888 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066: Bootstrap starting.
I20260924 04:36:11.969298 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:11.961655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.0.98.62:41289)
0924 04:36:11.962274 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:11.962289 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:11.962323 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:36:11.963031 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:11.963049 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:11.963445 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:11.963776 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:11.964180 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.965130 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.966100 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:11.966625 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:11.966785 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:11.967636 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:11.967641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:11.967652 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:11.968059 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:11.968114 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:11.968133 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:11.968232 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:11.968346 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:36:11.968588 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20260924 04:36:11.970144 5793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:11.970826 5793 log.cc:824] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:11.971829 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:36:11.972829 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066: No bootstrap required, opened a new log
I20260924 04:36:11.976109 5793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623084e1d3714df4b8e6415fb9ca6066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260924 04:36:11.976305 5793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:11.976358 5793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623084e1d3714df4b8e6415fb9ca6066, State: Initialized, Role: FOLLOWER
I20260924 04:36:11.976729 5793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623084e1d3714df4b8e6415fb9ca6066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260924 04:36:11.976841 5793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:11.976917 5793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:11.977025 5793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:11.977867 5793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623084e1d3714df4b8e6415fb9ca6066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260924 04:36:11.978184 5793 leader_election.cc:302] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 623084e1d3714df4b8e6415fb9ca6066; no voters:
I20260924 04:36:11.978451 5793 leader_election.cc:288] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:11.978567 5798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:11.978789 5798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [term 1 LEADER]: Becoming Leader. State: Replica: 623084e1d3714df4b8e6415fb9ca6066, State: Running, Role: LEADER
I20260924 04:36:11.979094 5798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623084e1d3714df4b8e6415fb9ca6066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } }
I20260924 04:36:11.979645 5793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:11.980235 5800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 623084e1d3714df4b8e6415fb9ca6066. Latest consensus state: current_term: 1 leader_uuid: "623084e1d3714df4b8e6415fb9ca6066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623084e1d3714df4b8e6415fb9ca6066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } } }
I20260924 04:36:11.980372 5800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:11.980691 5799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "623084e1d3714df4b8e6415fb9ca6066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623084e1d3714df4b8e6415fb9ca6066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41289 } } }
I20260924 04:36:11.980791 5799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:11.981181 5805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:11.984107 5805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:11.990021 5805 catalog_manager.cc:1347] Generated new cluster ID: 878c37d20ef5475280015442e8a021d8
I20260924 04:36:11.990087 5805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:12.015839 5805 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:12.016769 5805 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:12.027034 5805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 623084e1d3714df4b8e6415fb9ca6066: Generated new TSK 0
I20260924 04:36:12.027746 5805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:36:12.048007 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:12.043222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.0.98.62:41289)
0924 04:36:12.043344 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:12.043349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:12.043546 (+ 197us) server_negotiation.cc:407] Connection header received
0924 04:36:12.043703 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:12.043707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:12.043759 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:12.043865 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:36:12.044112 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.044638 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.045857 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.046070 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.046155 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:12.046442 (+ 287us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:36:12.047647 (+ 1205us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:36:12.047871 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 04:36:12.048076 5796 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 04:36:12.048125 5822 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:12.043159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41289 (local address 127.0.0.1:41402)
0924 04:36:12.043433 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:12.043447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:12.043565 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:12.043879 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:12.043882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:12.043891 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:36:12.044007 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:12.044013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.044737 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.044741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:12.045747 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:12.045756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.046162 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.046184 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:12.046271 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:12.046297 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:36:12.047802 (+ 1505us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:36:12.047939 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260924 04:36:12.048929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 5726
E20260924 04:36:12.061167 5823 webserver.cc:533] Webserver: sslize failed:
E20260924 04:36:12.061259 5823 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T04:36:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T04:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:12Z Disabled control of system clock
I20260924 04:36:12.096884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37251
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37251
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/testchainca.cert with env {}
W20260924 04:36:12.202768 5827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:12.203040 5827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:12.203106 5827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:12.207067 5827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:12.207139 5827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:12.207165 5827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:12.207202 5827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:12.207242 5827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:36:12.207278 5827 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:36:12.207314 5827 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 04:36:12.212424 5827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:12.213477 5827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:12.214438 5827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:12.222645 5835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:12.222622 5832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:12.222615 5833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:12.222915 5827 server_base.cc:1034] running on GCE node
I20260924 04:36:12.223403 5827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:12.224390 5827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:12.225579 5827 hybrid_clock.cc:648] HybridClock initialized: now 1790224572225565 us; error 35 us; skew 500 ppm
I20260924 04:36:12.227280 5827 webserver.cc:460] Webserver started at http://127.0.98.62:43825/ using document root <none> and password file <none>
I20260924 04:36:12.227833 5827 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:12.227891 5827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:12.228112 5827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:12.230087 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "27d03cca59ba43c3b4a9604f52d05746"
format_stamp: "Formatted at 2026-09-24 04:36:12 on dist-test-slave-jlnt"
server_key: "69ecbd5020b973597cffc08d9bb82478"
server_key_iv: "295611a2a4b161d684a64750c8c880d5"
server_key_version: "encryptionkey@0"
I20260924 04:36:12.230633 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "27d03cca59ba43c3b4a9604f52d05746"
format_stamp: "Formatted at 2026-09-24 04:36:12 on dist-test-slave-jlnt"
server_key: "69ecbd5020b973597cffc08d9bb82478"
server_key_iv: "295611a2a4b161d684a64750c8c880d5"
server_key_version: "encryptionkey@0"
I20260924 04:36:12.234107 5827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:12.236405 5841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:12.237694 5827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:12.237828 5827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "27d03cca59ba43c3b4a9604f52d05746"
format_stamp: "Formatted at 2026-09-24 04:36:12 on dist-test-slave-jlnt"
server_key: "69ecbd5020b973597cffc08d9bb82478"
server_key_iv: "295611a2a4b161d684a64750c8c880d5"
server_key_version: "encryptionkey@0"
I20260924 04:36:12.237933 5827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:12.256968 5827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:12.257617 5827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:12.257795 5827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:12.264320 5827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37251
I20260924 04:36:12.264389 5893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37251 every 8 connection(s)
I20260924 04:36:12.265245 5827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:12.268167 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:12.272037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 5827
I20260924 04:36:12.272178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:12.272465 392 external_mini_cluster.cc:1428] Setting key 43c6977a0a93597356d5eaa7b1920e52
I20260924 04:36:12.273697 5894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746: Bootstrap starting.
I20260924 04:36:12.275789 5894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:12.276559 5894 log.cc:824] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:12.278692 5894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746: No bootstrap required, opened a new log
I20260924 04:36:12.281823 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:12.273899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.0.98.62:37251)
0924 04:36:12.274474 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:12.274490 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:12.274524 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:36:12.275115 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:12.275131 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:12.275499 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:12.275766 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:12.276161 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.277355 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.278519 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.279027 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.279150 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:12.279983 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:12.279987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:12.280000 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:12.280374 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:12.280426 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:12.280446 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:12.280522 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:12.280706 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:36:12.281079 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":124,"threads_started":1}
I20260924 04:36:12.282518 5894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d03cca59ba43c3b4a9604f52d05746" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } }
I20260924 04:36:12.282764 5894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:12.282822 5894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d03cca59ba43c3b4a9604f52d05746, State: Initialized, Role: FOLLOWER
I20260924 04:36:12.283253 5894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d03cca59ba43c3b4a9604f52d05746" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } }
I20260924 04:36:12.283397 5894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:12.283459 5894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:12.283588 5894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:12.284621 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:36:12.284606 5894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d03cca59ba43c3b4a9604f52d05746" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } }
I20260924 04:36:12.284929 5894 leader_election.cc:302] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27d03cca59ba43c3b4a9604f52d05746; no voters:
I20260924 04:36:12.285287 5894 leader_election.cc:288] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:12.285385 5899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:12.285602 5899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [term 1 LEADER]: Becoming Leader. State: Replica: 27d03cca59ba43c3b4a9604f52d05746, State: Running, Role: LEADER
I20260924 04:36:12.285936 5899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d03cca59ba43c3b4a9604f52d05746" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } }
I20260924 04:36:12.286494 5894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:12.287587 5900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27d03cca59ba43c3b4a9604f52d05746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d03cca59ba43c3b4a9604f52d05746" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } }
I20260924 04:36:12.287715 5900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:12.287998 5901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27d03cca59ba43c3b4a9604f52d05746. Latest consensus state: current_term: 1 leader_uuid: "27d03cca59ba43c3b4a9604f52d05746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d03cca59ba43c3b4a9604f52d05746" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37251 } } }
I20260924 04:36:12.288134 5901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:12.288831 5908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:12.291312 5908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:12.296440 5908 catalog_manager.cc:1347] Generated new cluster ID: aa287f05fe954e28a30a369c39891651
I20260924 04:36:12.296505 5908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:12.311338 5908 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:12.312532 5908 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:12.320441 5908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27d03cca59ba43c3b4a9604f52d05746: Generated new TSK 0
I20260924 04:36:12.321085 5908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:36:12.350600 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:12.327972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.0.98.62:37251)
0924 04:36:12.328105 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:12.328110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:12.328272 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:36:12.328453 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:12.328456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:12.328508 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:12.328611 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:36:12.328856 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.329451 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.330338 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.330573 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.330680 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:12.330970 (+ 290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:36:12.350168 (+ 19198us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:36:12.350453 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60948: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 04:36:12.350684 5897 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60948: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 04:36:12.350747 5923 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:12.327884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37251 (local address 127.0.0.1:60948)
0924 04:36:12.328183 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:12.328195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:12.328317 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:12.328622 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:12.328626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:12.328633 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:36:12.328750 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:12.328755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.329597 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.329604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:12.330224 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:12.330232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:12.330676 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:12.330696 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:12.330791 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:12.330812 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:36:12.350356 (+ 19544us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:36:12.350534 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37251: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":104,"threads_started":1}
I20260924 04:36:12.351130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 5827
2026-09-24T04:36:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (280 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:12 dist-test-slave-jlnt krb5kdc[5935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:12 dist-test-slave-jlnt krb5kdc[5935](info): set up 2 sockets
Sep 24 04:36:12 dist-test-slave-jlnt krb5kdc[5935](info): commencing operation
krb5kdc: starting...
W20260924 04:36:13.625414 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:13 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:36:13.744793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40693
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:13.850811 5948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:13.851089 5948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:13.851162 5948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:13.855110 5948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:13.855190 5948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:13.855229 5948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:13.855274 5948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:13.855309 5948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:13.860410 5948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:13.861465 5948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:13.862437 5948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:13.870779 5953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:13.870764 5954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:13.870831 5948 server_base.cc:1034] running on GCE node
W20260924 04:36:13.870764 5956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:13.871549 5948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:13.872579 5948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:13.873754 5948 hybrid_clock.cc:648] HybridClock initialized: now 1790224573873740 us; error 27 us; skew 500 ppm
Sep 24 04:36:13 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:13.876713 5948 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:36:13.877575 5948 webserver.cc:460] Webserver started at http://127.0.98.62:37027/ using document root <none> and password file <none>
I20260924 04:36:13.878063 5948 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:13.878120 5948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:13.878275 5948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:13.880249 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "50901c1659214e85b71a9bbea84e4d97"
format_stamp: "Formatted at 2026-09-24 04:36:13 on dist-test-slave-jlnt"
server_key: "8fcd864c91b02b65dc8d8f99511100a1"
server_key_iv: "b2305d549f7a576651ce7ac699d4222c"
server_key_version: "encryptionkey@0"
I20260924 04:36:13.880750 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "50901c1659214e85b71a9bbea84e4d97"
format_stamp: "Formatted at 2026-09-24 04:36:13 on dist-test-slave-jlnt"
server_key: "8fcd864c91b02b65dc8d8f99511100a1"
server_key_iv: "b2305d549f7a576651ce7ac699d4222c"
server_key_version: "encryptionkey@0"
I20260924 04:36:13.884284 5948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:36:13.886607 5963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:13.887817 5948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:13.887943 5948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "50901c1659214e85b71a9bbea84e4d97"
format_stamp: "Formatted at 2026-09-24 04:36:13 on dist-test-slave-jlnt"
server_key: "8fcd864c91b02b65dc8d8f99511100a1"
server_key_iv: "b2305d549f7a576651ce7ac699d4222c"
server_key_version: "encryptionkey@0"
I20260924 04:36:13.888047 5948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:13.923156 5948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:13.926570 5948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:13.926772 5948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:13.933398 5948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40693
I20260924 04:36:13.933672 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40693 every 8 connection(s)
I20260924 04:36:13.934353 5948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:13.936991 6016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:13.940570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 5948
I20260924 04:36:13.940704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:13.941000 392 external_mini_cluster.cc:1428] Setting key a5e7ac66bb9a014ff6a7a5b37b3b2a8b
I20260924 04:36:13.942704 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97: Bootstrap starting.
I20260924 04:36:13.945065 6016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:13.945799 6016 log.cc:824] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:13.947767 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97: No bootstrap required, opened a new log
Sep 24 04:36:13 dist-test-slave-jlnt krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:13.951826 6016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50901c1659214e85b71a9bbea84e4d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40693 } }
I20260924 04:36:13.952055 6016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:13.952139 6016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50901c1659214e85b71a9bbea84e4d97, State: Initialized, Role: FOLLOWER
I20260924 04:36:13.952656 6016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50901c1659214e85b71a9bbea84e4d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40693 } }
I20260924 04:36:13.952814 6016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:13.952885 6016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:13.953001 6016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:13.954030 6016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50901c1659214e85b71a9bbea84e4d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40693 } }
I20260924 04:36:13.954365 6016 leader_election.cc:302] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50901c1659214e85b71a9bbea84e4d97; no voters:
I20260924 04:36:13.954694 6016 leader_election.cc:288] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:13.954780 6021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:13.954943 6021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [term 1 LEADER]: Becoming Leader. State: Replica: 50901c1659214e85b71a9bbea84e4d97, State: Running, Role: LEADER
I20260924 04:36:13.955088 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:13.942386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.0.98.62:40693)
0924 04:36:13.942802 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:13.942824 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:13.942866 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:36:13.943459 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:13.943475 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:13.943795 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:13.944062 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:13.944941 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:13.946036 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:13.947068 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:13.947507 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:13.947623 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:13.950292 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:13.950297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:13.950309 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:13.950334 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:13.952028 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:13.952562 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:13.952567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:13.952571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:13.952677 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:13.952958 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:13.952963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:13.952965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:13.953429 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:13.953582 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:13.953871 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 04:36:13.954169 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":120,"threads_started":1}
I20260924 04:36:13.955278 6021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50901c1659214e85b71a9bbea84e4d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40693 } }
I20260924 04:36:13.955925 6016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:13.956864 6023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50901c1659214e85b71a9bbea84e4d97. Latest consensus state: current_term: 1 leader_uuid: "50901c1659214e85b71a9bbea84e4d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50901c1659214e85b71a9bbea84e4d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40693 } } }
I20260924 04:36:13.957268 6023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:13.957162 6022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50901c1659214e85b71a9bbea84e4d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50901c1659214e85b71a9bbea84e4d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40693 } } }
I20260924 04:36:13.957583 6022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97 [sys.catalog]: This master's current role is: LEADER
W20260924 04:36:13.960520 6037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:36:13.960611 6037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:36:13.960687 6030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:13.961536 6030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:13.966243 6030 catalog_manager.cc:1347] Generated new cluster ID: f9a0dbcc35af40afb074116d5b61bf2b
I20260924 04:36:13.966302 6030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:13.986176 6030 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:13.987149 6030 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:13.993729 6030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50901c1659214e85b71a9bbea84e4d97: Generated new TSK 0
I20260924 04:36:13.994542 6030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:36:14.024076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40693
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:14.132154 6042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:14.132391 6042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:14.132432 6042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:14.136299 6042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:14.136351 6042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:14.136406 6042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:14.141402 6042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:14.142444 6042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:14.143374 6042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:14.152730 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:14.152863 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:14.152899 6042 server_base.cc:1034] running on GCE node
W20260924 04:36:14.152863 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:14.153513 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:14.154098 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:14.155272 6042 hybrid_clock.cc:648] HybridClock initialized: now 1790224574155258 us; error 29 us; skew 500 ppm
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.158072 6042 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:36:14.158896 6042 webserver.cc:460] Webserver started at http://127.0.98.1:35991/ using document root <none> and password file <none>
I20260924 04:36:14.159432 6042 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:14.159489 6042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:14.159643 6042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:14.161561 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "44c0ae810f9340f3b1504b762ba4f047"
format_stamp: "Formatted at 2026-09-24 04:36:14 on dist-test-slave-jlnt"
server_key: "2890a9c81bcde292db907366726e6ea1"
server_key_iv: "5b61b921caac69fdebafb393a4d4d150"
server_key_version: "encryptionkey@0"
I20260924 04:36:14.162045 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "44c0ae810f9340f3b1504b762ba4f047"
format_stamp: "Formatted at 2026-09-24 04:36:14 on dist-test-slave-jlnt"
server_key: "2890a9c81bcde292db907366726e6ea1"
server_key_iv: "5b61b921caac69fdebafb393a4d4d150"
server_key_version: "encryptionkey@0"
I20260924 04:36:14.165483 6042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:36:14.167618 6057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:14.168730 6042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:14.168839 6042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "44c0ae810f9340f3b1504b762ba4f047"
format_stamp: "Formatted at 2026-09-24 04:36:14 on dist-test-slave-jlnt"
server_key: "2890a9c81bcde292db907366726e6ea1"
server_key_iv: "5b61b921caac69fdebafb393a4d4d150"
server_key_version: "encryptionkey@0"
I20260924 04:36:14.168933 6042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:14.182875 6042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:14.185520 6042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:14.185660 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:14.186128 6042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:14.186851 6042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:14.186895 6042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:14.186932 6042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:14.186947 6042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:14.195647 6042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43275
I20260924 04:36:14.195668 6170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43275 every 8 connection(s)
I20260924 04:36:14.196475 6042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:14.199738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 6042
I20260924 04:36:14.199888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:14.200243 392 external_mini_cluster.cc:1428] Setting key 02ba83e231e7c8b8f1ba594c5844448b
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:36:14.210664 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:14.198174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58513 (local address 127.0.98.62:40693)
0924 04:36:14.198314 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:14.198319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:14.199175 (+ 856us) server_negotiation.cc:407] Connection header received
0924 04:36:14.200878 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:14.200881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:14.200936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:14.201017 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:14.202500 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.203038 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.204400 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.204625 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.204731 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:14.207099 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:14.207103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:14.207105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:14.207134 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:14.208626 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.209128 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.209131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.209154 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.209238 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.209578 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.209583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.209585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.209778 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:14.209884 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:14.210354 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 04:36:14.210478 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:36:14.211413 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:14.198573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40693 (local address 127.0.98.1:58513)
0924 04:36:14.199015 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:14.199065 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:14.200695 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:14.201233 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:14.201242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:14.201668 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:14.202328 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:14.202350 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.203361 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.203365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:14.204276 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:14.204285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.204767 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.204772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:14.204942 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:14.205542 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:14.205558 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:14.206943 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:14.208739 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:14.208744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:14.208758 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:14.209037 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:14.209335 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:14.209339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:14.209341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:14.209476 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:14.209882 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:14.209889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:14.210135 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 04:36:14.210316 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 04:36:14.210624 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:36:14.212646 6171 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40693
I20260924 04:36:14.212878 6171 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:14.213511 6171 heartbeater.cc:511] Master 127.0.98.62:40693 requested a full tablet report, sending...
I20260924 04:36:14.215054 5980 ts_manager.cc:194] Registered new tserver with Master: 44c0ae810f9340f3b1504b762ba4f047 (127.0.98.1:43275)
I20260924 04:36:14.216195 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:36:14.227483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40693
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:14.333940 6176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:14.334226 6176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:14.334300 6176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:14.338271 6176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:14.338356 6176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:14.338449 6176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:14.343493 6176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:14.344485 6176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:14.345468 6176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:36:14.354838 6176 server_base.cc:1034] running on GCE node
W20260924 04:36:14.354754 6181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:14.354756 6182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:14.354754 6184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:14.355381 6176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:14.356001 6176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:14.357179 6176 hybrid_clock.cc:648] HybridClock initialized: now 1790224574357161 us; error 27 us; skew 500 ppm
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.359846 6176 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:36:14.360634 6176 webserver.cc:460] Webserver started at http://127.0.98.2:41379/ using document root <none> and password file <none>
I20260924 04:36:14.361238 6176 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:14.361320 6176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:14.361515 6176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:14.363431 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "e00eefbdebb046d3992dc96651de626c"
format_stamp: "Formatted at 2026-09-24 04:36:14 on dist-test-slave-jlnt"
server_key: "05583248b2f2207db3bf6f4f151bf1ef"
server_key_iv: "3c8ef6715596a5800f42f8edab780181"
server_key_version: "encryptionkey@0"
I20260924 04:36:14.363988 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e00eefbdebb046d3992dc96651de626c"
format_stamp: "Formatted at 2026-09-24 04:36:14 on dist-test-slave-jlnt"
server_key: "05583248b2f2207db3bf6f4f151bf1ef"
server_key_iv: "3c8ef6715596a5800f42f8edab780181"
server_key_version: "encryptionkey@0"
I20260924 04:36:14.367449 6176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:14.369760 6191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:14.370847 6176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:14.370966 6176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "e00eefbdebb046d3992dc96651de626c"
format_stamp: "Formatted at 2026-09-24 04:36:14 on dist-test-slave-jlnt"
server_key: "05583248b2f2207db3bf6f4f151bf1ef"
server_key_iv: "3c8ef6715596a5800f42f8edab780181"
server_key_version: "encryptionkey@0"
I20260924 04:36:14.371062 6176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:14.383823 6176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:14.386780 6176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:14.386968 6176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:14.387497 6176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:14.388347 6176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:14.388409 6176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:14.388471 6176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:14.388516 6176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:14.397038 6176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42167
I20260924 04:36:14.397073 6304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42167 every 8 connection(s)
I20260924 04:36:14.397902 6176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:14.403116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 6176
I20260924 04:36:14.403254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:14.403532 392 external_mini_cluster.cc:1428] Setting key 2f72186298d80a57999545653f31dbc5
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:14.412250 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:14.399542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46101 (local address 127.0.98.62:40693)
0924 04:36:14.399668 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:14.399672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:14.400415 (+ 743us) server_negotiation.cc:407] Connection header received
0924 04:36:14.401823 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:14.401826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:14.401883 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:14.401979 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:14.403214 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.403950 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.405403 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.405675 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.405760 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:14.408567 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:14.408571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:14.408574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:14.408596 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:14.410060 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.410693 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.410696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.410697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.410768 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.411152 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.411155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.411156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.411317 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:14.411390 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:14.412025 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 04:36:14.412135 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:36:14.413347 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:14.399865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40693 (local address 127.0.98.2:46101)
0924 04:36:14.400272 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:14.400313 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:14.401629 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:14.402143 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:14.402150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:14.402488 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:14.403011 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:14.403029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.404098 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.404101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:14.405291 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:14.405304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.405820 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.405827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:14.406062 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:14.406739 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:14.406754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:14.408414 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:14.410174 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:14.410182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:14.410200 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:14.410592 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:14.410862 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:14.410868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:14.410871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:14.411054 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:14.411416 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:14.411425 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:14.411731 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 04:36:14.411939 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 04:36:14.412267 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:36:14.414438 6305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40693
I20260924 04:36:14.414645 6305 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:14.415060 6305 heartbeater.cc:511] Master 127.0.98.62:40693 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:36:14.416065 5980 ts_manager.cc:194] Registered new tserver with Master: e00eefbdebb046d3992dc96651de626c (127.0.98.2:42167)
I20260924 04:36:14.416782 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:36:14.432046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40693
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:14.556226 6310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:14.556501 6310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:14.556550 6310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:14.560683 6310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:14.560738 6310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:14.560794 6310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:14.565987 6310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:14.567245 6310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:14.568238 6310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:14.577608 6315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:14.577608 6318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:14.577620 6316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:14.577853 6310 server_base.cc:1034] running on GCE node
I20260924 04:36:14.578210 6310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:14.578809 6310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:14.579977 6310 hybrid_clock.cc:648] HybridClock initialized: now 1790224574579958 us; error 30 us; skew 500 ppm
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.582670 6310 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:36:14.583498 6310 webserver.cc:460] Webserver started at http://127.0.98.3:43771/ using document root <none> and password file <none>
I20260924 04:36:14.584003 6310 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:14.584060 6310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:14.584219 6310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:14.586220 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "b9011f1d5931467e823b85e6987c7c4a"
format_stamp: "Formatted at 2026-09-24 04:36:14 on dist-test-slave-jlnt"
server_key: "38f38e5da0783bff20ff1e61157dae09"
server_key_iv: "689890e227ada66b43a119e3768c7525"
server_key_version: "encryptionkey@0"
I20260924 04:36:14.586743 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b9011f1d5931467e823b85e6987c7c4a"
format_stamp: "Formatted at 2026-09-24 04:36:14 on dist-test-slave-jlnt"
server_key: "38f38e5da0783bff20ff1e61157dae09"
server_key_iv: "689890e227ada66b43a119e3768c7525"
server_key_version: "encryptionkey@0"
I20260924 04:36:14.590277 6310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:14.592711 6325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:14.593943 6310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:36:14.594072 6310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "b9011f1d5931467e823b85e6987c7c4a"
format_stamp: "Formatted at 2026-09-24 04:36:14 on dist-test-slave-jlnt"
server_key: "38f38e5da0783bff20ff1e61157dae09"
server_key_iv: "689890e227ada66b43a119e3768c7525"
server_key_version: "encryptionkey@0"
I20260924 04:36:14.594174 6310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:14.607985 6310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:14.611064 6310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:14.611263 6310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:14.611860 6310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:14.612705 6310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:14.612807 6310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:14.612902 6310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:14.612950 6310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:14.621611 6310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43679
I20260924 04:36:14.621650 6438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43679 every 8 connection(s)
I20260924 04:36:14.622474 6310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:14.627885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 6310
I20260924 04:36:14.628016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:14.628322 392 external_mini_cluster.cc:1428] Setting key 12d9a4778a5211d50ad5344b3f578423
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:14.635917 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:14.624242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54959 (local address 127.0.98.62:40693)
0924 04:36:14.624379 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:14.624383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:14.625294 (+ 911us) server_negotiation.cc:407] Connection header received
0924 04:36:14.626678 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:14.626681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:14.626735 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:14.626838 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:14.628000 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.628624 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.630029 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.630337 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.630450 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:14.632848 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:14.632851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:14.632853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:14.632875 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:14.634258 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.634725 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.634728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.634729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.634800 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.635064 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.635066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.635067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.635222 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:14.635295 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:14.635714 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 04:36:14.635804 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:36:14.636686 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:14.624611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40693 (local address 127.0.98.3:54959)
0924 04:36:14.625091 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:14.625182 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:14.626468 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:14.627006 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:14.627012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:14.627390 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:14.627859 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:14.627872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.628745 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.628749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:14.629887 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:14.629898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.630486 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.630489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:14.630661 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:14.631226 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:14.631241 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:14.632689 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:14.634356 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:14.634360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:14.634371 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:14.634616 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:14.634869 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:14.634871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:14.634872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:14.634994 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:14.635299 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:14.635304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:14.635512 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 04:36:14.635676 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:36:14.635903 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
I20260924 04:36:14.637699 6439 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40693
I20260924 04:36:14.637913 6439 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:14.638358 6439 heartbeater.cc:511] Master 127.0.98.62:40693 requested a full tablet report, sending...
I20260924 04:36:14.639281 5980 ts_manager.cc:194] Registered new tserver with Master: b9011f1d5931467e823b85e6987c7c4a (127.0.98.3:43679)
I20260924 04:36:14.639828 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54959
I20260924 04:36:14.642011 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/test-admin.keytab.
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.657869 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 04:36:14.660634 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:14.659197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.98.1:43275)
0924 04:36:14.659540 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:14.659551 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:14.659578 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:36:14.659629 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:14.659633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:14.659832 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:14.659951 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:14.660434 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33952: BlockingRecv error: recv got EOF from 127.0.0.1:33952 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":142,"threads_started":1}
W20260924 04:36:14.660717 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.661820 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.662704 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:36:14.672374 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:14.663477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.98.1:43275)
0924 04:36:14.663596 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:14.663600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:14.663737 (+ 137us) server_negotiation.cc:407] Connection header received
0924 04:36:14.663879 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:14.663882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:14.663929 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:14.664007 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:14.664506 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.665371 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.666194 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.666563 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.666675 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:14.669063 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:14.669071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:14.669084 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:14.669105 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:14.670621 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.671047 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.671053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.671060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.671140 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.671422 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.671425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.671427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.671663 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:14.671873 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:14.672055 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 04:36:14.672209 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:36:14.676013 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:14.675165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.0.98.1:43275)
0924 04:36:14.675256 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:14.675259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:14.675342 (+ 83us) server_negotiation.cc:407] Connection header received
0924 04:36:14.675477 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:14.675479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:14.675509 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:14.675574 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:14.675908 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33968: BlockingRecv error: recv got EOF from 127.0.0.1:33968 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 04:36:14.676172 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.677213 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.678131 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:36:14.686532 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:14.678888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33978 (local address 127.0.98.1:43275)
0924 04:36:14.679014 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:14.679017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:14.679270 (+ 253us) server_negotiation.cc:407] Connection header received
0924 04:36:14.679352 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:14.679354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:14.679385 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:14.679449 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:14.679870 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.680354 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.681099 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.681398 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.681490 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:14.683600 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:14.683603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:14.683605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:14.683627 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:14.684948 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.685423 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.685425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.685427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.685499 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.685785 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.685787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.685794 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.685937 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:14.686029 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:14.686249 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 04:36:14.686331 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 04:36:14.688959 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:14.688199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.98.1:43275)
0924 04:36:14.688315 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:14.688318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:14.688356 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:36:14.688455 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:14.688457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:14.688487 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:14.688543 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:14.688882 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33980: BlockingRecv error: recv got EOF from 127.0.0.1:33980 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:36:14.689325 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790224545298817-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.690248 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:14.691045 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:36:14.699443 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:14.691824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33994 (local address 127.0.98.1:43275)
0924 04:36:14.691941 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:14.691944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:14.691959 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:14.692075 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:14.692080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:14.692119 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:14.692189 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:14.692740 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.693264 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.694029 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:14.694240 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:14.694325 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:14.696461 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:14.696464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:14.696466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:14.696486 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:14.697960 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.698315 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.698317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.698318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.698383 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:14.698612 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:14.698616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:14.698617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:14.698826 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:14.698952 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:14.699179 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 04:36:14.699284 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:36:14.700323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 6042
I20260924 04:36:14.706243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 6176
I20260924 04:36:14.711272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 6310
I20260924 04:36:14.716619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 5948
2026-09-24T04:36:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2125 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[6459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[6459](info): set up 2 sockets
Sep 24 04:36:14 dist-test-slave-jlnt krb5kdc[6459](info): commencing operation
krb5kdc: starting...
W20260924 04:36:15.765697 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:15 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:36:15.884389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33081
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:15.990870 6472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:15.991146 6472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:15.991190 6472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:15.995209 6472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:15.995268 6472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:15.995288 6472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:15.995304 6472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:15.995362 6472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:16.000550 6472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:16.001600 6472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:16.002624 6472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:16.011137 6480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:16.011155 6477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:16.011231 6472 server_base.cc:1034] running on GCE node
W20260924 04:36:16.011147 6478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:16.011961 6472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:16.013087 6472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:16.014293 6472 hybrid_clock.cc:648] HybridClock initialized: now 1790224576014279 us; error 26 us; skew 500 ppm
Sep 24 04:36:16 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:16.026348 6472 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:36:16.027381 6472 webserver.cc:460] Webserver started at http://127.0.98.62:44029/ using document root <none> and password file <none>
I20260924 04:36:16.027968 6472 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:16.028071 6472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:16.028307 6472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:16.030459 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "27d96944a97f400ab082a7cf95b97dd7"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "10ca3bf3b58a0d4f1497cd2d9e043260"
server_key_iv: "765317498f5b88230f254fceff9431dd"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.031078 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "27d96944a97f400ab082a7cf95b97dd7"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "10ca3bf3b58a0d4f1497cd2d9e043260"
server_key_iv: "765317498f5b88230f254fceff9431dd"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.035075 6472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:36:16.037648 6487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.038899 6472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:16.039038 6472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "27d96944a97f400ab082a7cf95b97dd7"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "10ca3bf3b58a0d4f1497cd2d9e043260"
server_key_iv: "765317498f5b88230f254fceff9431dd"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.039147 6472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:16.068511 6472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:16.071493 6472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:16.071708 6472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:16.078646 6472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33081
I20260924 04:36:16.078933 6539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33081 every 8 connection(s)
I20260924 04:36:16.079643 6472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:16.080292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 6472
I20260924 04:36:16.080447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:16.080749 392 external_mini_cluster.cc:1428] Setting key 3ae011d99fa027653ebde707b42e184a
I20260924 04:36:16.082887 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:16.089458 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: Bootstrap starting.
Sep 24 04:36:16 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:16.092357 6540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:16.093240 6540 log.cc:824] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:16.095538 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: No bootstrap required, opened a new log
I20260924 04:36:16.095669 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:16.082070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.0.98.62:33081)
0924 04:36:16.082473 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:16.082489 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:16.082525 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:36:16.083241 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:16.083260 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:16.083582 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:16.083869 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:16.084734 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.085748 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.086892 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.087348 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.087463 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:16.090360 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:16.090366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:16.090382 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:16.090414 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:16.092531 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:16.092994 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:16.093002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:16.093010 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:16.093175 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:16.093531 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:16.093534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:16.093536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:16.093977 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:16.094173 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:16.094464 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 04:36:16.094756 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":134,"threads_started":1}
I20260924 04:36:16.099375 6540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } }
I20260924 04:36:16.099586 6540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:16.099622 6540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d96944a97f400ab082a7cf95b97dd7, State: Initialized, Role: FOLLOWER
I20260924 04:36:16.099975 6540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } }
I20260924 04:36:16.100107 6540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:16.100169 6540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:16.100262 6540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:16.101318 6540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } }
I20260924 04:36:16.101643 6540 leader_election.cc:302] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27d96944a97f400ab082a7cf95b97dd7; no voters:
I20260924 04:36:16.101946 6540 leader_election.cc:288] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:16.102162 6545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:16.102455 6545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 27d96944a97f400ab082a7cf95b97dd7, State: Running, Role: LEADER
I20260924 04:36:16.102747 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } }
I20260924 04:36:16.103291 6540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:16.104372 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27d96944a97f400ab082a7cf95b97dd7. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } }
I20260924 04:36:16.104487 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:16.104836 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } }
I20260924 04:36:16.104985 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:16.105523 6554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:16.108098 6554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:16.113365 6554 catalog_manager.cc:1347] Generated new cluster ID: a699fc3a67f940ae8e60c856aea58cb6
I20260924 04:36:16.113425 6554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:16.133610 6554 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:16.134927 6554 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:16.151714 6554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: Generated new TSK 0
I20260924 04:36:16.152426 6554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:36:16.189525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33081
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:16.326694 6566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:16.326938 6566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:16.326980 6566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:16.330958 6566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:16.331013 6566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:16.331107 6566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:16.336262 6566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:16.337240 6566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:16.338310 6566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:16.347772 6571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:16.347894 6572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:16.347918 6574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:16.347978 6566 server_base.cc:1034] running on GCE node
I20260924 04:36:16.348433 6566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:16.349072 6566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:16.350287 6566 hybrid_clock.cc:648] HybridClock initialized: now 1790224576350265 us; error 33 us; skew 500 ppm
Sep 24 04:36:16 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:16.353017 6566 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:36:16.353885 6566 webserver.cc:460] Webserver started at http://127.0.98.1:46659/ using document root <none> and password file <none>
I20260924 04:36:16.354429 6566 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:16.354518 6566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:16.354745 6566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:16.356768 6566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "e4029b6d160547fdb897b6ee50f4b9ec"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "7e0484ca44ca8f44a77224fcff95ed1f"
server_key_iv: "ba4c7704acdc08e8f50af107dba8f6da"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.357406 6566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e4029b6d160547fdb897b6ee50f4b9ec"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "7e0484ca44ca8f44a77224fcff95ed1f"
server_key_iv: "ba4c7704acdc08e8f50af107dba8f6da"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.360854 6566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:36:16.363260 6581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.364470 6566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:16.364599 6566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "e4029b6d160547fdb897b6ee50f4b9ec"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "7e0484ca44ca8f44a77224fcff95ed1f"
server_key_iv: "ba4c7704acdc08e8f50af107dba8f6da"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.364715 6566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:16.386936 6566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:16.390251 6566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:16.390457 6566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:16.391045 6566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:16.391997 6566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:16.392082 6566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.392155 6566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:16.392206 6566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.400938 6566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40885
I20260924 04:36:16.400960 6694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40885 every 8 connection(s)
I20260924 04:36:16.401808 6566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:16.407001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 6566
I20260924 04:36:16.407130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:16.407405 392 external_mini_cluster.cc:1428] Setting key 542eaee06ee0a56e8d580ed6d5bfc735
Sep 24 04:36:16 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:16.417905 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:16.403742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33807 (local address 127.0.98.62:33081)
0924 04:36:16.403899 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:16.403903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:16.404856 (+ 953us) server_negotiation.cc:407] Connection header received
0924 04:36:16.406392 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:16.406396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:16.406462 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:16.406583 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:16.408085 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.408947 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.410302 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.410579 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.410701 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:16.413640 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:16.413646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:16.413652 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:16.413687 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:16.415536 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:16.416078 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:16.416084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:16.416088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:16.416193 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:16.416540 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:16.416553 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:16.416555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:16.416815 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:16.416936 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:16.417580 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 04:36:16.417726 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:36:16.418645 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:16.404210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.1:33807)
0924 04:36:16.404703 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:16.404749 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:16.406160 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:16.406760 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:16.406767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:16.407259 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:16.407869 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:16.407890 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.409098 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.409102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:16.410163 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:16.410173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.410738 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.410743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:16.410925 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:16.411613 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:16.411640 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:16.413458 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:16.415680 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:16.415684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:16.415697 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:16.415949 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:16.416298 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:16.416300 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:16.416302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:16.416430 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:16.416951 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:16.416957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:16.417274 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 04:36:16.417457 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 04:36:16.417727 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260924 04:36:16.420015 6695 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33081
I20260924 04:36:16.420272 6695 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 04:36:16.421015 6695 heartbeater.cc:511] Master 127.0.98.62:33081 requested a full tablet report, sending...
I20260924 04:36:16.422544 6504 ts_manager.cc:194] Registered new tserver with Master: e4029b6d160547fdb897b6ee50f4b9ec (127.0.98.1:40885)
I20260924 04:36:16.424064 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:36:16.436519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33081
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:16.545354 6700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:16.545601 6700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:16.545679 6700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:16.549893 6700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:16.549966 6700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:16.550053 6700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:16.556325 6700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:16.557358 6700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:16.558326 6700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:16.567700 6708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:16.567719 6705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:16.567907 6700 server_base.cc:1034] running on GCE node
W20260924 04:36:16.567831 6706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:16.568473 6700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:16.569175 6700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:16.570355 6700 hybrid_clock.cc:648] HybridClock initialized: now 1790224576570297 us; error 69 us; skew 500 ppm
Sep 24 04:36:16 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:16.573308 6700 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:36:16.574244 6700 webserver.cc:460] Webserver started at http://127.0.98.2:46417/ using document root <none> and password file <none>
I20260924 04:36:16.574837 6700 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:16.574899 6700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:16.575132 6700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:16.577215 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "b574eca291e34af69b610f262ea5dc21"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "e9e9fbe5ab0648a282c3dc5168a2ab75"
server_key_iv: "b42f51ad2b20d768371f30d57211fb3f"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.577793 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b574eca291e34af69b610f262ea5dc21"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "e9e9fbe5ab0648a282c3dc5168a2ab75"
server_key_iv: "b42f51ad2b20d768371f30d57211fb3f"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.581422 6700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:36:16.583786 6715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.585002 6700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:16.585094 6700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "b574eca291e34af69b610f262ea5dc21"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "e9e9fbe5ab0648a282c3dc5168a2ab75"
server_key_iv: "b42f51ad2b20d768371f30d57211fb3f"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.585280 6700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:16.605336 6700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:16.608654 6700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:16.608870 6700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:16.609545 6700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:16.610355 6700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:16.610423 6700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.610493 6700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:16.610543 6700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.619048 6700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33661
I20260924 04:36:16.619072 6828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33661 every 8 connection(s)
I20260924 04:36:16.619917 6700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:16.622375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 6700
I20260924 04:36:16.622483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:16.622808 392 external_mini_cluster.cc:1428] Setting key c3c3d1cf812c6288a8e9f67b4288815f
Sep 24 04:36:16 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:36:16.634851 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:16.621631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39187 (local address 127.0.98.62:33081)
0924 04:36:16.621762 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:16.621766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:16.622707 (+ 941us) server_negotiation.cc:407] Connection header received
0924 04:36:16.624303 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:16.624306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:16.624346 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:16.624413 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:16.626034 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.626896 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.628020 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.628275 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.628388 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:16.631311 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:16.631314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:16.631316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:16.631338 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:16.632736 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:16.633364 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:16.633366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:16.633367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:16.633437 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:16.633841 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:16.633844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:16.633845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:16.634006 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:16.634079 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:16.634537 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 04:36:16.634632 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:36:16.635722 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:16.622022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.2:39187)
0924 04:36:16.622516 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:16.622575 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:16.624066 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:16.624599 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:16.624609 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:16.625175 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:16.625859 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:16.625882 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.627036 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.627039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:16.627912 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:16.627921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.628425 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.628429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:16.628595 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:16.629283 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:16.629300 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:16.631148 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:16.632852 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:16.632861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:16.632881 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:16.633270 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:16.633530 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:16.633536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:16.633541 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:16.633739 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:16.634098 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:16.634103 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:16.634332 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 04:36:16.634480 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 04:36:16.634781 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 04:36:16.636930 6829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33081
I20260924 04:36:16.637228 6829 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:16.637718 6829 heartbeater.cc:511] Master 127.0.98.62:33081 requested a full tablet report, sending...
I20260924 04:36:16.638955 6504 ts_manager.cc:194] Registered new tserver with Master: b574eca291e34af69b610f262ea5dc21 (127.0.98.2:33661)
I20260924 04:36:16.639575 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:36:16.652071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33081
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:16.760069 6834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:16.760334 6834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:16.760375 6834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:16.764628 6834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:16.764688 6834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:16.764746 6834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:16.770128 6834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:16.771133 6834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:16.772128 6834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:16.781466 6842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:16.781452 6840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:16.781450 6839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:16.781728 6834 server_base.cc:1034] running on GCE node
I20260924 04:36:16.782135 6834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:16.782719 6834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:16.783876 6834 hybrid_clock.cc:648] HybridClock initialized: now 1790224576783858 us; error 40 us; skew 500 ppm
Sep 24 04:36:16 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:16.786566 6834 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:36:16.787381 6834 webserver.cc:460] Webserver started at http://127.0.98.3:43409/ using document root <none> and password file <none>
I20260924 04:36:16.787918 6834 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:16.787976 6834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:16.788201 6834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:16.790233 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c83f256d94d401bb7d3b788ab2e248e"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "eef705d382f0448bfe8bda29699ea98d"
server_key_iv: "433c0b3a3263aaa101c555f759a92a8a"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.790791 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c83f256d94d401bb7d3b788ab2e248e"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "eef705d382f0448bfe8bda29699ea98d"
server_key_iv: "433c0b3a3263aaa101c555f759a92a8a"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.794432 6834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:16.796640 6849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.797881 6834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:16.797971 6834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "9c83f256d94d401bb7d3b788ab2e248e"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "eef705d382f0448bfe8bda29699ea98d"
server_key_iv: "433c0b3a3263aaa101c555f759a92a8a"
server_key_version: "encryptionkey@0"
I20260924 04:36:16.798053 6834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:16.822575 6834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:16.825450 6834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:16.825600 6834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:16.826113 6834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:16.826843 6834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:16.826889 6834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.826925 6834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:16.826941 6834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:16.835132 6834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40723
I20260924 04:36:16.835156 6962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40723 every 8 connection(s)
I20260924 04:36:16.835963 6834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:16.837944 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 6834
I20260924 04:36:16.838053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:16.838331 392 external_mini_cluster.cc:1428] Setting key c4dd2ff9a8da6ea1d4a1f00343b483a7
Sep 24 04:36:16 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:16.848595 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:16.837754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59469 (local address 127.0.98.62:33081)
0924 04:36:16.837899 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:16.837903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:16.838793 (+ 890us) server_negotiation.cc:407] Connection header received
0924 04:36:16.840137 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:16.840141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:16.840193 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:16.840282 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:16.841396 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.841904 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.843022 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.843229 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.843305 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:16.845637 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:16.845640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:16.845642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:16.845664 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:16.846883 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:16.847333 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:16.847335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:16.847337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:16.847419 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:16.847732 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:16.847734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:16.847735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:16.847909 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:16.847983 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:16.848384 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 04:36:16.848477 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:36:16.849377 6965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:16.838204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.3:59469)
0924 04:36:16.838652 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:16.838701 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:16.839948 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:16.840469 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:16.840476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:16.840821 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:16.841280 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:16.841293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.842007 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.842009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:16.842924 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:16.842934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:16.843346 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:16.843348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:16.843547 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:16.844081 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:16.844096 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:16.845453 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:16.846979 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:16.846983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:16.846993 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:16.847230 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:16.847504 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:16.847506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:16.847507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:16.847645 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:16.847992 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:16.847996 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:16.848198 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 04:36:16.848341 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 04:36:16.848561 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":130,"threads_started":1}
I20260924 04:36:16.850479 6963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33081
I20260924 04:36:16.850708 6963 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:16.851233 6963 heartbeater.cc:511] Master 127.0.98.62:33081 requested a full tablet report, sending...
I20260924 04:36:16.852213 6503 ts_manager.cc:194] Registered new tserver with Master: 9c83f256d94d401bb7d3b788ab2e248e (127.0.98.3:40723)
I20260924 04:36:16.852783 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59469
I20260924 04:36:16.863862 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 04:36:16.889950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35263
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33081,127.0.98.61:35263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:16.997375 6968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:16.997677 6968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:16.997756 6968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:17.001701 6968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:17.001771 6968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:17.001806 6968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:17.001861 6968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:17.001899 6968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:17.007341 6968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33081,127.0.98.61:35263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
W20260924 04:36:17.007584 6968 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:33081,127.0.98.61:35263'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 04:36:17.008545 6968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:17.009663 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:17.017697 6976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:17.017714 6973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:17.017728 6974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:17.018199 6968 server_base.cc:1034] running on GCE node
I20260924 04:36:17.018738 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:17.019785 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:17.020960 6968 hybrid_clock.cc:648] HybridClock initialized: now 1790224577020949 us; error 34 us; skew 500 ppm
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:17.023617 6968 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:36:17.024389 6968 webserver.cc:460] Webserver started at http://127.0.98.61:35333/ using document root <none> and password file <none>
I20260924 04:36:17.024900 6968 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:17.024953 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:17.025103 6968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:17.027029 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data/instance:
uuid: "a83a227cb46d4afb8fc2faf1d7907a98"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "783e7ddaa0bb5e10a02a5db5f912d010"
server_key_iv: "ef5724757ad10602af735e1681926374"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.027514 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal/instance:
uuid: "a83a227cb46d4afb8fc2faf1d7907a98"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "783e7ddaa0bb5e10a02a5db5f912d010"
server_key_iv: "ef5724757ad10602af735e1681926374"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.031186 6968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:17.033437 6983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.034628 6968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:36:17.034729 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal
uuid: "a83a227cb46d4afb8fc2faf1d7907a98"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "783e7ddaa0bb5e10a02a5db5f912d010"
server_key_iv: "ef5724757ad10602af735e1681926374"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.034806 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:17.051218 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:17.054069 6968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:17.054211 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:17.060725 6968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35263
I20260924 04:36:17.060737 7035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35263 every 8 connection(s)
I20260924 04:36:17.061686 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:36:17.064294 7036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:17.065527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 6968
I20260924 04:36:17.068372 7036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:17.081123 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.070083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.0.98.62:33081)
0924 04:36:17.070237 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.070243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.070499 (+ 256us) server_negotiation.cc:407] Connection header received
0924 04:36:17.070632 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.070635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.070683 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.070768 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.071415 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.072210 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.073428 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.073729 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.073854 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.076603 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.076606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.076608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.076652 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.078195 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.078924 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.078928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.078930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.079047 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.079634 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.079637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.079639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.079970 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.080090 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.080747 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.080911 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:36:17.082986 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.070039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:35970)
0924 04:36:17.070369 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.070415 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.070528 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.070813 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.070817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.071134 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.071283 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.071292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.072337 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.072340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.073326 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.073333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.073896 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.073900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.073993 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.074500 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.074524 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.076452 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.078475 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.078484 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.078494 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.078798 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.079206 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.079209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.079211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.079475 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.080196 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.080202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.080421 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.081432 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.081866 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"spinlock_wait_cycles":3456,"thread_start_us":114,"threads_started":1}
I20260924 04:36:17.082993 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.067036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.0.98.61:35263)
0924 04:36:17.067722 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.067746 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.067783 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:36:17.068392 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.068408 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.068834 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.069235 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.070045 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.071297 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.072493 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.073180 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.073338 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.076073 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.076079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.076105 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.076127 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.078195 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.079318 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.079325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.079327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.079448 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.079790 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.079794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.079799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.080298 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.080494 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.081647 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.081969 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":436,"thread_start_us":126,"threads_started":1}
I20260924 04:36:17.087321 7036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:36:17.097198 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.088057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.0.98.61:35263)
0924 04:36:17.088352 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.088355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.088364 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:36:17.088388 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.088390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.088420 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.088477 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.089059 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.090291 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.091477 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.091828 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.091926 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.094246 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.094250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.094252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.094278 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.095628 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.096033 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.096039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.096040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.096118 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.096381 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.096384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.096385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.096547 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.096656 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.096879 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.097020 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":249}
I20260924 04:36:17.097198 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.087856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.0.1:49578)
0924 04:36:17.088147 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.088159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.088244 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.088530 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.088534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.088727 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.088865 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.088873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.090407 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.090412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.091357 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.091368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.091945 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.091950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.092087 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.092582 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.092601 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.094120 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.095750 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.095754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.095757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.095940 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.096211 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.096214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.096216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.096292 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.096644 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.096647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.096725 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.096838 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.096969 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260924 04:36:17.100674 7036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: Bootstrap starting.
I20260924 04:36:17.103732 7036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:17.104692 7036 log.cc:824] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:17.106957 7036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: No bootstrap required, opened a new log
I20260924 04:36:17.111716 7036 raft_consensus.cc:348] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } }
I20260924 04:36:17.111974 7036 raft_consensus.cc:374] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:17.112046 7036 raft_consensus.cc:729] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83a227cb46d4afb8fc2faf1d7907a98, State: Initialized, Role: FOLLOWER
I20260924 04:36:17.112558 7036 consensus_queue.cc:260] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } }
I20260924 04:36:17.113938 7036 sys_catalog.cc:564] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:36:17.119036 7053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:36:17.119102 7053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:36:17.120453 6968 master_runner.cc:428] Detected that this master a83a227cb46d4afb8fc2faf1d7907a98 is joining an existing cluster
I20260924 04:36:17.120503 6968 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:35263
I20260924 04:36:17.121356 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } } }
I20260924 04:36:17.121601 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:36:17.132649 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.122899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.0.98.61:35263)
0924 04:36:17.123021 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.123027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.123619 (+ 592us) server_negotiation.cc:407] Connection header received
0924 04:36:17.123713 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.123716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.123758 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.123838 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.124293 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.125043 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.126362 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.126745 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.126828 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.128649 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.128653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.128656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.128679 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.131025 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.131459 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.131462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.131464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.131531 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.131796 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.131799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.131801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.131993 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.132061 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.132363 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.132481 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:36:17.133277 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.123183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.0.98.62:33081)
0924 04:36:17.123303 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.123308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.123322 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:36:17.123440 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.123443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.123489 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.123543 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.124245 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.125048 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.126341 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.126693 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.126770 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.128306 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.128310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.128313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.128341 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.130224 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.131822 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.131827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.131829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.131917 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.132252 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.132256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.132259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.132487 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.132563 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.132934 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.133074 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:36:17.133667 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.122990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.0.1:49588)
0924 04:36:17.123529 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.123543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.123648 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.123860 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.123862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.124089 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.124205 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.124211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.125172 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.125175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.126227 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.126241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.126982 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.126986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.127093 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.127787 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.127804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.128177 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.131137 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.131140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.131142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.131365 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.131620 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.131623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.131625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.131709 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.132119 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.132121 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.132209 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.133172 (+ 963us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.133301 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":384,"thread_start_us":91,"threads_started":1}
I20260924 04:36:17.133908 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.122808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:35974)
0924 04:36:17.123086 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.123100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.123193 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.123714 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.123716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.123977 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.124100 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.124119 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.125212 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.125215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.126209 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.126218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.126935 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.126938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.127041 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.127629 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.127643 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.128160 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.131412 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.131417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.131419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.131690 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.132035 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.132039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.132041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.132143 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.132642 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.132646 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.132742 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.133488 (+ 746us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.133641 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":4,"thread_start_us":106,"threads_started":1}
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:36:17.149501 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.139399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.0.98.61:35263)
0924 04:36:17.139512 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.139516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.139528 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:36:17.139587 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.139590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.139645 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.139718 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.140502 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.141282 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.142496 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.142820 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.142917 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.145687 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.145693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.145697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.145726 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.147495 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.148059 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.148063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.148066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.148151 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.148459 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.148463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.148466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.148679 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.148778 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.149182 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.149298 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:36:17.149685 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.138675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.0.1:49596)
0924 04:36:17.139006 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.139055 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.139163 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.139839 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.139845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.140259 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.140375 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.140382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.141412 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.141415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.142393 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.142403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.142930 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.142933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.143049 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.143575 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.143590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.145539 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.147597 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.147608 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.147617 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.147950 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.148250 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.148255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.148259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.148364 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.148793 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.148799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.148931 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.149093 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.149327 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 04:36:17.151288 6503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:35263: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 35263
}
attrs {
promote: true
}
}
dest_uuid: "27d96944a97f400ab082a7cf95b97dd7"
cas_config_opid_index: -1
I20260924 04:36:17.151928 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } }
I20260924 04:36:17.153712 7065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a83a227cb46d4afb8fc2faf1d7907a98 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } } }
I20260924 04:36:17.154297 7010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:17.156533 7065 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:35263
I20260924 04:36:17.156626 7010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Refusing update from remote peer 27d96944a97f400ab082a7cf95b97dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 04:36:17.156858 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } } }
I20260924 04:36:17.156965 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:17.157353 6968 master.cc:537] Master@127.0.98.61:35263 shutting down...
I20260924 04:36:17.157559 7064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:36:17.159806 6491 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 04:36:17.161039 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 -> Peer a83a227cb46d4afb8fc2faf1d7907a98 (127.0.98.61:35263): Couldn't send request to peer a83a227cb46d4afb8fc2faf1d7907a98. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:36:17.162237 6968 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:17.162528 6968 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:17.162636 6968 tablet_replica.cc:330] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260924 04:36:17.180527 6968 master.cc:552] Master@127.0.98.61:35263 shutdown complete.
I20260924 04:36:17.180595 6968 master_runner.cc:305] Clearing existing system tablet
I20260924 04:36:17.180820 6968 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:17.180888 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:17.182143 6968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:17.183302 7070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.183493 6968 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:17.183624 6968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:17.183673 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal
uuid: "a83a227cb46d4afb8fc2faf1d7907a98"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "783e7ddaa0bb5e10a02a5db5f912d010"
server_key_iv: "ef5724757ad10602af735e1681926374"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.185545 6968 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 04:36:17.187647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37173
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33081,127.0.98.61:35263,127.0.98.60:37173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:36:17.188234 6968 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:36:17.188323 6968 log.cc:1196] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 04:36:17.188563 6968 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: Deleting consensus metadata
I20260924 04:36:17.188802 6968 master_runner.cc:315] Copying system tablet from 127.0.98.62:33081
I20260924 04:36:17.190110 6968 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33081
I20260924 04:36:17.198299 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.190805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.0.98.62:33081)
0924 04:36:17.190921 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.190928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.191025 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:36:17.191189 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.191192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.191246 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.191332 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.191776 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.192366 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.193570 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.193848 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.193911 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.195113 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.195117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.195119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.195155 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.196608 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.197043 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.197048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.197049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.197129 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.197489 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.197493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.197494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.197673 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.197777 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.198036 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.198153 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:36:17.198307 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.190694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:35982)
0924 04:36:17.190917 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.190933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.191085 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.191348 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.191351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.191555 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.191658 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.191664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.192483 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.192488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.193404 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.193413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.194042 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.194044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.194114 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.194653 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.194664 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.194969 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.196739 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.196742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.196744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.196942 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.197306 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.197309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.197311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.197393 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.197767 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.197770 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.197868 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.197972 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.198107 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260924 04:36:17.198823 6524 tablet_copy_service.cc:140] P 27d96944a97f400ab082a7cf95b97dd7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a83a227cb46d4afb8fc2faf1d7907a98 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35982)
I20260924 04:36:17.198964 6524 tablet_copy_service.cc:161] P 27d96944a97f400ab082a7cf95b97dd7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a83a227cb46d4afb8fc2faf1d7907a98 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35982: session id = a83a227cb46d4afb8fc2faf1d7907a98-00000000000000000000000000000000
I20260924 04:36:17.200228 6524 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:36:17.201675 6968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:36:17.203722 6968 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: tablet copy: Starting download of 0 data blocks...
I20260924 04:36:17.203938 6968 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: tablet copy: Starting download of 1 WAL segments...
I20260924 04:36:17.205588 6968 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:36:17.207039 6524 tablet_copy_service.cc:342] P 27d96944a97f400ab082a7cf95b97dd7: Request end of tablet copy session a83a227cb46d4afb8fc2faf1d7907a98-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35982
I20260924 04:36:17.207168 6524 tablet_copy_service.cc:433] P 27d96944a97f400ab082a7cf95b97dd7: ending tablet copy session a83a227cb46d4afb8fc2faf1d7907a98-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a83a227cb46d4afb8fc2faf1d7907a98
I20260924 04:36:17.214210 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:17.215778 7078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:17.215920 7081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:17.216393 6968 server_base.cc:1034] running on GCE node
W20260924 04:36:17.216487 7079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:17.216689 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:17.216984 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:17.218123 6968 hybrid_clock.cc:648] HybridClock initialized: now 1790224577218125 us; error 53 us; skew 500 ppm
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:17.219532 6968 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:36:17.219969 6968 webserver.cc:460] Webserver started at http://127.0.98.61:40963/ using document root <none> and password file <none>
I20260924 04:36:17.220152 6968 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:17.220206 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:17.221369 6968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:17.222244 7088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.222467 6968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:17.222556 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal
uuid: "a83a227cb46d4afb8fc2faf1d7907a98"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "783e7ddaa0bb5e10a02a5db5f912d010"
server_key_iv: "ef5724757ad10602af735e1681926374"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.222710 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:17.228947 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:17.230837 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:17.235507 6968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35263
I20260924 04:36:17.235600 7140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35263 every 8 connection(s)
I20260924 04:36:17.236027 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:36:17.237469 7141 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:36:17.238230 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: Bootstrap starting.
I20260924 04:36:17.246050 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:36:17.246323 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: Bootstrap complete.
I20260924 04:36:17.246764 7141 raft_consensus.cc:348] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } }
I20260924 04:36:17.246893 7141 raft_consensus.cc:729] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a83a227cb46d4afb8fc2faf1d7907a98, State: Initialized, Role: LEARNER
I20260924 04:36:17.247004 7141 consensus_queue.cc:260] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } }
I20260924 04:36:17.247311 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } } }
I20260924 04:36:17.247455 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:36:17.247457 7141 sys_catalog.cc:564] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:17.250098 7155 catalog_manager.cc:1259] Loaded cluster ID: a699fc3a67f940ae8e60c856aea58cb6
I20260924 04:36:17.250381 7155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: loading cluster ID for follower catalog manager: success
I20260924 04:36:17.251866 7155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: acquiring CA information for follower catalog manager: success
I20260924 04:36:17.252887 7155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 04:36:17.333841 7071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:17.334144 7071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:17.334223 7071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:17.338259 7071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:17.338335 7071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:17.338373 7071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:17.338409 7071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:17.338440 7071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:17.343747 7071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33081,127.0.98.61:35263,127.0.98.60:37173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:17.344755 7071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:17.345849 7071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:17.354295 7163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:17.354317 7071 server_base.cc:1034] running on GCE node
W20260924 04:36:17.354328 7161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:17.354503 7160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:17.355041 7071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:17.356055 7071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:17.357210 7071 hybrid_clock.cc:648] HybridClock initialized: now 1790224577357178 us; error 47 us; skew 500 ppm
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:17.359771 7071 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:36:17.360575 7071 webserver.cc:460] Webserver started at http://127.0.98.60:45023/ using document root <none> and password file <none>
I20260924 04:36:17.361096 7071 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:17.361212 7071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:17.361434 7071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:17.363379 7071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data/instance:
uuid: "61fad00dd08f42808ded7d0901de51d1"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "e6853252f3946a95600ed2ff011819dc"
server_key_iv: "dff7ebe37b7694de62c37009669dbb83"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.363926 7071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal/instance:
uuid: "61fad00dd08f42808ded7d0901de51d1"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "e6853252f3946a95600ed2ff011819dc"
server_key_iv: "dff7ebe37b7694de62c37009669dbb83"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.367547 7071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 04:36:17.369837 7170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.370982 7071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 04:36:17.371111 7071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal
uuid: "61fad00dd08f42808ded7d0901de51d1"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "e6853252f3946a95600ed2ff011819dc"
server_key_iv: "dff7ebe37b7694de62c37009669dbb83"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.371218 7071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:17.426437 6695 heartbeater.cc:503] Master 127.0.98.62:33081 was elected leader, sending a full tablet report...
I20260924 04:36:17.444835 7071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:17.447953 7071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:17.448168 7071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:17.455013 7071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37173
I20260924 04:36:17.455322 7222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37173 every 8 connection(s)
I20260924 04:36:17.456035 7071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:36:17.458792 7223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:17.462460 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:36:17.466179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 7071
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:36:17.476815 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.464358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.0.98.62:33081)
0924 04:36:17.464506 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.464511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.465298 (+ 787us) server_negotiation.cc:407] Connection header received
0924 04:36:17.466320 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.466326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.466384 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.466452 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.467784 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.468373 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.469888 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.470230 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.470340 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.472739 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.472744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.472748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.472782 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.474316 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.474968 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.474972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.474974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.475092 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.475576 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.475580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.475581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.475820 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.475901 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.476510 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.476650 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:36:17.477639 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.464686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:35994)
0924 04:36:17.465097 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.465192 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.466052 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.466666 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.466677 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.467094 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.467632 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.467652 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.468501 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.468504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.469757 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.469767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.470373 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.470376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.470514 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.471058 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.471077 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.472591 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.474432 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.474442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.474452 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.474853 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.475231 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.475234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.475236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.475459 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.475926 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.475934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.476234 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.476409 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.476729 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
I20260924 04:36:17.478132 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.467030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.0.98.60:37173)
0924 04:36:17.467362 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.467372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.467390 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:36:17.467423 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.467426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.467571 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.467744 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.468475 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.469828 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.470813 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.471288 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.471393 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.474201 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.474206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.474215 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.474234 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.476116 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.476640 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.476645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.476647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.476728 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.477072 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.477076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.477078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.477387 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.477635 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.477792 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.477932 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":230,"thread_start_us":138,"threads_started":1}
I20260924 04:36:17.478940 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:36:17.487244 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.479431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.0.1:49604)
0924 04:36:17.479604 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.479619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.479693 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.480074 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.480076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.480232 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.480313 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.480317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.481329 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.481332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.481903 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.481911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.482393 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.482395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.482472 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.482900 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.482910 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.484072 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.486008 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.486010 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.486011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.486208 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.486446 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.486448 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.486449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.486517 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.486903 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.486906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.486973 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.487051 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.487147 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 04:36:17.487288 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.479546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.0.98.61:35263)
0924 04:36:17.479827 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.479832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.479848 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:36:17.479891 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.479894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.479949 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.480050 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.480418 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.481218 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.482027 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.482272 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.482360 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.484222 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.484225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.484227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.484250 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.485908 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.486295 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.486297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.486299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.486371 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.486639 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.486642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.486643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.486819 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.486898 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.487081 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.487175 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260924 04:36:17.487975 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:36:17.495136 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.488446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.0.1:40572)
0924 04:36:17.488578 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.488591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.488653 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.488842 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.488844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.488969 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.489043 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.489048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.489818 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.489821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.490353 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.490360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.490767 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.490770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.490832 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.491209 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.491222 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.492316 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.493921 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.493923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.493925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.494125 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.494349 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.494351 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.494353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.494423 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.494747 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.494749 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.494817 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.494903 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.495003 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:36:17.495160 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.488558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.0.98.60:37173)
0924 04:36:17.488671 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.488675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.488693 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:36:17.488719 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.488721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.488750 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.488810 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.489196 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.489717 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.490447 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.490675 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.490746 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.492429 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.492432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.492433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.492452 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.493828 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.494207 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.494209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.494210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.494274 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.494495 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.494497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.494498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.494666 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.494738 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.494935 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.495052 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":53}
I20260924 04:36:17.498247 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Bootstrap starting.
I20260924 04:36:17.500403 7223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:17.501278 7223 log.cc:824] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:17.503111 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: No bootstrap required, opened a new log
I20260924 04:36:17.506354 7223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } }
I20260924 04:36:17.506558 7223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:17.506639 7223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61fad00dd08f42808ded7d0901de51d1, State: Initialized, Role: FOLLOWER
I20260924 04:36:17.507035 7223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } }
I20260924 04:36:17.507465 7229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } } }
I20260924 04:36:17.507611 7229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:36:17.508126 7223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:36:17.512809 7240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:36:17.512894 7240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:36:17.515830 7071 master_runner.cc:428] Detected that this master 61fad00dd08f42808ded7d0901de51d1 is joining an existing cluster
I20260924 04:36:17.515898 7071 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:37173
I20260924 04:36:17.531283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 6566
I20260924 04:36:17.537374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40885
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33081
--webserver_port=46659
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33081,127.0.98.61:35263
--tserver_master_addrs=127.0.98.62:33081,127.0.98.61:35263,127.0.98.60:37173 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:36:17.547600 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.517887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.0.98.62:33081)
0924 04:36:17.518037 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.518041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.518633 (+ 592us) server_negotiation.cc:407] Connection header received
0924 04:36:17.518722 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.518728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.518785 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.518856 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.519676 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.520485 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.522427 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.523008 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.523110 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.542243 (+ 19133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.542250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.542255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.542293 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.545420 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.546011 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.546015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.546017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.546102 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.546381 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.546385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.546387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.546623 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.546716 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.547317 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.547428 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:36:17.548090 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.517831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:36006)
0924 04:36:17.518519 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.518543 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.518638 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.518974 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.518979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.519287 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.519543 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.519551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.520627 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.520631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.522320 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.522325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.523250 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.523253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.523339 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.524088 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.524109 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.542013 (+ 17904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.545583 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.545588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.545592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.545908 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.546204 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.546207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.546209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.546288 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.546757 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.546761 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.546881 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.546976 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.547123 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":112,"spinlock_wait_cycles":15104,"thread_start_us":111,"threads_started":1}
I20260924 04:36:17.548372 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.518303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.0.98.60:37173)
0924 04:36:17.518383 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.518387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.518879 (+ 492us) server_negotiation.cc:407] Connection header received
0924 04:36:17.519013 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.519016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.519051 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.519095 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.519774 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.520500 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.521702 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.522086 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.522190 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.542158 (+ 19968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.542163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.542165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.542191 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.544288 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.546108 (+ 1820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.546114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.546118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.546861 (+ 743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.547199 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.547206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.547210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.547477 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.547585 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.547991 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.548120 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":23}
I20260924 04:36:17.549044 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.518247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.0.1:40580)
0924 04:36:17.518815 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.518826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.518926 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.519211 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.519214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.519532 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.519655 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.519663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.520634 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.520637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.521582 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.521592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.522212 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.522216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.522348 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.523241 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.523255 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.541998 (+ 18743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.544764 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.544767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.544769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.545992 (+ 1223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.546987 (+ 995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.546990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.546992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.547094 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.547637 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.547642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.547745 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.548615 (+ 870us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.548779 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":21,"thread_start_us":80,"threads_started":1}
I20260924 04:36:17.550585 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.517904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.0.98.61:35263)
0924 04:36:17.518035 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.518040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.518632 (+ 592us) server_negotiation.cc:407] Connection header received
0924 04:36:17.518802 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.518807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.518860 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.518905 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.519676 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.520478 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.522630 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.522984 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.523098 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.543322 (+ 20224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.543326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.543329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.543361 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.545655 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.549027 (+ 3372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.549031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.549032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.549128 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.549627 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.549630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.549632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.549831 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.549899 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.550242 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.550407 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:36:17.551342 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.518152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.0.1:49606)
0924 04:36:17.518541 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.518551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.518700 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.519028 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.519032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.519322 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.519542 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.519551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.520627 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.520631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.522497 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.522505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.523138 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.523142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.523241 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.523685 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.523702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.543155 (+ 19453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.545778 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.545781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.545783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.548904 (+ 3121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.549397 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.549399 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.549401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.549508 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.549967 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.549970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.550067 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.551011 (+ 944us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.551181 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":30,"thread_start_us":92,"threads_started":1}
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:36:17.561338 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.552595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.0.98.60:37173)
0924 04:36:17.552688 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.552692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.552704 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:36:17.552770 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.552772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.552818 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.552898 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.553368 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.554118 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.555247 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.555567 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.555655 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.557979 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.557991 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.557996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.558020 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.559620 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.560117 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.560120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.560122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.560205 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.560487 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.560490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.560492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.560686 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.560815 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.561073 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.561200 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:36:17.561393 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.552377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.0.1:40582)
0924 04:36:17.552598 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.552611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.552682 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.552926 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.552929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.553110 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.553240 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.553247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.554250 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.554253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.555114 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.555124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.555758 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.555762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.555854 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.556306 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.556320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.557847 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.559732 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.559736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.559738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.559981 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.560303 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.560305 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.560307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.560394 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.560802 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.560805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.560898 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.561003 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.561237 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 04:36:17.562640 6503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:37173: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "61fad00dd08f42808ded7d0901de51d1"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 37173
}
attrs {
promote: true
}
}
dest_uuid: "27d96944a97f400ab082a7cf95b97dd7"
cas_config_opid_index: 5
I20260924 04:36:17.563246 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } }
I20260924 04:36:17.564254 7064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 61fad00dd08f42808ded7d0901de51d1 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.565748 7064 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:37173
I20260924 04:36:17.565909 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.565990 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:17.566327 7197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:17.566819 7071 master.cc:537] Master@127.0.98.60:37173 shutting down...
I20260924 04:36:17.567714 7197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 FOLLOWER]: Refusing update from remote peer 27d96944a97f400ab082a7cf95b97dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:36:17.568528 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:36:17.570247 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 -> Peer 61fad00dd08f42808ded7d0901de51d1 (127.0.98.60:37173): Couldn't send request to peer 61fad00dd08f42808ded7d0901de51d1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:36:17.572966 7071 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:17.573102 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.565631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.0.98.61:35263)
0924 04:36:17.568235 (+ 2604us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.568240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.568490 (+ 250us) server_negotiation.cc:407] Connection header received
0924 04:36:17.568539 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.568542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.568598 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.568674 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:17.569023 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.569812 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.571424 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.572034 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.572130 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.572280 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.572750 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.572825 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2506}
I20260924 04:36:17.573261 7071 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:17.573364 7071 tablet_replica.cc:330] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: stopping tablet replica
I20260924 04:36:17.574386 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.565412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.0.1:49610)
0924 04:36:17.568322 (+ 2910us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.568336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.568415 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.568813 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.568816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.568830 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:17.568910 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.568915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.569939 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.569943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.571182 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.571202 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.572341 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.572344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.572439 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.572478 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.572548 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.574294 (+ 1746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2843}
I20260924 04:36:17.574949 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 LEARNER]: Refusing update from remote peer 27d96944a97f400ab082a7cf95b97dd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:36:17.575462 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 04:36:17.577368 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 61fad00dd08f42808ded7d0901de51d1 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.579602 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27d96944a97f400ab082a7cf95b97dd7. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.579723 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:36:17.579810 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: true } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.579872 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:36:17.581455 7065 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: attempting to promote NON_VOTER a83a227cb46d4afb8fc2faf1d7907a98 to VOTER
I20260924 04:36:17.581907 7065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } }
I20260924 04:36:17.583431 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.583060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.0.1:40584)
0924 04:36:17.583293 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.583358 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260924 04:36:17.583986 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 LEARNER]: Refusing update from remote peer 27d96944a97f400ab082a7cf95b97dd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 04:36:17.584347 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.584054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.0.1:40600)
0924 04:36:17.584245 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.584289 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260924 04:36:17.584393 7252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 04:36:17.584513 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 -> Peer 61fad00dd08f42808ded7d0901de51d1 (127.0.98.60:37173): Couldn't send request to peer 61fad00dd08f42808ded7d0901de51d1. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:36:17.585929 7064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a83a227cb46d4afb8fc2faf1d7907a98 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.587395 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.587507 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:17.588255 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.588367 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:17.589107 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a83a227cb46d4afb8fc2faf1d7907a98 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.591024 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.591138 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:36:17.591542 7071 master.cc:552] Master@127.0.98.60:37173 shutdown complete.
I20260924 04:36:17.591585 7071 master_runner.cc:305] Clearing existing system tablet
I20260924 04:36:17.591838 7071 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:17.591884 7071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:17.593150 7071 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:36:17.594003 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.594439 7071 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:17.594691 7071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:17.594769 7071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal
uuid: "61fad00dd08f42808ded7d0901de51d1"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "e6853252f3946a95600ed2ff011819dc"
server_key_iv: "dff7ebe37b7694de62c37009669dbb83"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.596781 7071 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:17.599157 7071 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:36:17.599303 7071 log.cc:1196] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 04:36:17.599599 7071 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Deleting consensus metadata
I20260924 04:36:17.599798 7071 master_runner.cc:315] Copying system tablet from 127.0.98.62:33081
I20260924 04:36:17.600953 7071 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33081
I20260924 04:36:17.616568 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.601665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.0.98.62:33081)
0924 04:36:17.601780 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.601785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.608241 (+ 6456us) server_negotiation.cc:407] Connection header received
0924 04:36:17.608365 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.608369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.608425 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.608538 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.609122 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.610007 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.611206 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.611550 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.611655 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.613047 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.613051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.613055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.613080 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.614730 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.615248 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.615252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.615254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.615336 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.615620 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.615624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.615626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.615832 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.615901 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.616290 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.616404 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:36:17.617381 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.601575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:36014)
0924 04:36:17.608109 (+ 6534us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.608127 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.608274 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.608574 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.608577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.608795 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.609019 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.609027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.610147 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.610159 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.611084 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.611095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.611673 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.611677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.611770 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.612477 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.612492 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.612905 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.614850 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.614853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.614856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.615093 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.615442 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.615445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.615447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.615527 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.615970 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.615973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.616064 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.617002 (+ 938us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.617198 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6435,"thread_start_us":99,"threads_started":1}
I20260924 04:36:17.618031 6524 tablet_copy_service.cc:140] P 27d96944a97f400ab082a7cf95b97dd7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 61fad00dd08f42808ded7d0901de51d1 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36014)
I20260924 04:36:17.618121 6524 tablet_copy_service.cc:161] P 27d96944a97f400ab082a7cf95b97dd7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 61fad00dd08f42808ded7d0901de51d1 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36014: session id = 61fad00dd08f42808ded7d0901de51d1-00000000000000000000000000000000
I20260924 04:36:17.619186 6524 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:36:17.620512 7071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:36:17.622996 7071 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: tablet copy: Starting download of 0 data blocks...
I20260924 04:36:17.623260 7071 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: tablet copy: Starting download of 1 WAL segments...
I20260924 04:36:17.624714 7071 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:36:17.626432 6524 tablet_copy_service.cc:342] P 27d96944a97f400ab082a7cf95b97dd7: Request end of tablet copy session 61fad00dd08f42808ded7d0901de51d1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36014
I20260924 04:36:17.626506 6524 tablet_copy_service.cc:433] P 27d96944a97f400ab082a7cf95b97dd7: ending tablet copy session 61fad00dd08f42808ded7d0901de51d1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 61fad00dd08f42808ded7d0901de51d1
I20260924 04:36:17.633940 7071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:17.635630 7262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:17.635640 7265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:17.635953 7071 server_base.cc:1034] running on GCE node
W20260924 04:36:17.636235 7263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:17.636476 7071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:17.636768 7071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:17.637913 7071 hybrid_clock.cc:648] HybridClock initialized: now 1790224577637903 us; error 29 us; skew 500 ppm
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:17.639662 7071 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:36:17.640200 7071 webserver.cc:460] Webserver started at http://127.0.98.60:36151/ using document root <none> and password file <none>
I20260924 04:36:17.640416 7071 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:17.640466 7071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:17.641618 6829 heartbeater.cc:503] Master 127.0.98.62:33081 was elected leader, sending a full tablet report...
I20260924 04:36:17.645495 7071 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.002s
I20260924 04:36:17.646977 7272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.647218 7071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:36:17.647290 7071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal
uuid: "61fad00dd08f42808ded7d0901de51d1"
format_stamp: "Formatted at 2026-09-24 04:36:17 on dist-test-slave-jlnt"
server_key: "e6853252f3946a95600ed2ff011819dc"
server_key_iv: "dff7ebe37b7694de62c37009669dbb83"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.647361 7071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260924 04:36:17.673013 7249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:17.673333 7249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:17.673420 7249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:36:17.674474 7071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:17.676581 7071 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260924 04:36:17.678340 7249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:17.678403 7249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:17.678501 7249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:17.681025 7071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37173
I20260924 04:36:17.681299 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37173 every 8 connection(s)
I20260924 04:36:17.681600 7071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:36:17.683256 7328 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:36:17.684278 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Bootstrap starting.
I20260924 04:36:17.685969 7249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46659
--tserver_master_addrs=127.0.98.62:33081,127.0.98.61:35263,127.0.98.60:37173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:17.687146 7249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:17.688373 7249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:36:17.695140 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:36:17.695451 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Bootstrap complete.
I20260924 04:36:17.695923 7328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } }
I20260924 04:36:17.696059 7328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61fad00dd08f42808ded7d0901de51d1, State: Initialized, Role: LEARNER
I20260924 04:36:17.696180 7328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } }
I20260924 04:36:17.696470 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:17.696625 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:36:17.696630 7328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:17.699314 7347 catalog_manager.cc:1259] Loaded cluster ID: a699fc3a67f940ae8e60c856aea58cb6
I20260924 04:36:17.699430 7347 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: loading cluster ID for follower catalog manager: success
W20260924 04:36:17.700003 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:17.700845 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:17.700011 7335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:17.701529 7249 server_base.cc:1034] running on GCE node
I20260924 04:36:17.701937 7249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:17.702241 7347 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: acquiring CA information for follower catalog manager: success
I20260924 04:36:17.702693 7249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:17.703555 7347 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:36:17.703872 7249 hybrid_clock.cc:648] HybridClock initialized: now 1790224577703833 us; error 47 us; skew 500 ppm
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:17.706523 7249 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:36:17.707307 7249 webserver.cc:460] Webserver started at http://127.0.98.1:46659/ using document root <none> and password file <none>
I20260924 04:36:17.707870 7249 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:17.707942 7249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:17.712136 7249 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:17.714231 7354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.715400 7249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:17.715495 7249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "e4029b6d160547fdb897b6ee50f4b9ec"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "7e0484ca44ca8f44a77224fcff95ed1f"
server_key_iv: "ba4c7704acdc08e8f50af107dba8f6da"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.716012 7249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:17.725417 7249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:17.728258 7249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:17.728442 7249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:17.728958 7249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:17.729871 7249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:17.729918 7249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.729974 7249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:17.730005 7249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.739375 7249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40885
I20260924 04:36:17.739455 7467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40885 every 8 connection(s)
I20260924 04:36:17.740329 7249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:17.744287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 7249
I20260924 04:36:17.744503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 6700
I20260924 04:36:17.751317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33661
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33081
--webserver_port=46417
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33081,127.0.98.61:35263
--tserver_master_addrs=127.0.98.62:33081,127.0.98.61:35263,127.0.98.60:37173 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:36:17.758250 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.743614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44419 (local address 127.0.98.62:33081)
0924 04:36:17.743763 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.743768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.746720 (+ 2952us) server_negotiation.cc:407] Connection header received
0924 04:36:17.746810 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.746813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.746860 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.746939 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.747493 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.748310 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.749533 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.749861 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.749962 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.753797 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.753801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.753803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.753828 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.755874 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.756416 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.756419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.756422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.756497 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.756917 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.756920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.756922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.757123 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.757233 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.757960 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.758074 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:36:17.758415 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.742174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52483 (local address 127.0.98.60:37173)
0924 04:36:17.742463 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.742468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.743138 (+ 670us) server_negotiation.cc:407] Connection header received
0924 04:36:17.744449 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.744455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.744537 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.744612 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.746469 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.747652 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.749430 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.749761 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.749826 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.753783 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.753786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.753789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.753815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.755721 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.756409 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.756412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.756414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.756497 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.756917 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.756920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.756922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.757123 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.757233 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.757973 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.758145 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260924 04:36:17.759172 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.743312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.1:44419)
0924 04:36:17.746613 (+ 3301us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.746626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.746719 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.747022 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.747026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.747251 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.747388 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.747395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.748427 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.748430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.749411 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.749419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.750026 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.750030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.750297 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.751110 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.751133 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.753602 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.755981 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.755984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.755986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.756311 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.756599 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.756601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.756603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.756806 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.757286 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.757292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.757672 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.758377 (+ 705us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.758498 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3241,"mutex_wait_us":93,"spinlock_wait_cycles":16896,"thread_start_us":80,"threads_started":1}
I20260924 04:36:17.759172 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.742490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.98.1:52483)
0924 04:36:17.742968 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.743020 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.744150 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.745398 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.745407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.745784 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.746313 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.746326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.747813 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.747816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.749285 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.749295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.750021 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.750025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.750297 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.751271 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.751286 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.753602 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.755857 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.755863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.755878 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.756294 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.756597 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.756600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.756602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.756806 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.757286 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.757292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.757672 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.757888 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.758190 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":139,"spinlock_wait_cycles":17280,"thread_start_us":105,"threads_started":1}
I20260924 04:36:17.760501 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33081
I20260924 04:36:17.760704 7469 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:17.761278 7469 heartbeater.cc:511] Master 127.0.98.62:33081 requested a full tablet report, sending...
I20260924 04:36:17.760439 7468 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37173
I20260924 04:36:17.761896 7468 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:17.762059 7468 heartbeater.cc:511] Master 127.0.98.60:37173 requested a full tablet report, sending...
I20260924 04:36:17.762603 6503 ts_manager.cc:194] Re-registered known tserver with Master: e4029b6d160547fdb897b6ee50f4b9ec (127.0.98.1:40885)
I20260924 04:36:17.762823 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.742047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53435 (local address 127.0.98.61:35263)
0924 04:36:17.742176 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.742181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.743398 (+ 1217us) server_negotiation.cc:407] Connection header received
0924 04:36:17.744353 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.744357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.744418 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.744497 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.751491 (+ 6994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.752276 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.753465 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.753892 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.754033 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.756702 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.756709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.756716 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.756751 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.760865 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.761344 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.761350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.761354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.761453 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.761763 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.761769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.761771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.762050 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.762165 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.762574 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.762693 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:36:17.762936 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.742491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.98.1:53435)
0924 04:36:17.743296 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.743310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.744157 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.750797 (+ 6640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.750800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.750990 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.751346 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.751357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.752418 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.752424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.753336 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.753346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.754202 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.754208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.754328 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.754805 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.754824 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.756537 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.760977 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.760980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.760982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.761241 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.761559 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.761562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.761564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.761646 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.762223 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.762229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.762357 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.762481 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.762646 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":27,"spinlock_wait_cycles":2816,"thread_start_us":126,"threads_started":1}
I20260924 04:36:17.763367 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44419
I20260924 04:36:17.764618 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35263
I20260924 04:36:17.764705 7470 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:17.764838 7470 heartbeater.cc:511] Master 127.0.98.61:35263 requested a full tablet report, sending...
I20260924 04:36:17.764881 7292 ts_manager.cc:194] Registered new tserver with Master: e4029b6d160547fdb897b6ee50f4b9ec (127.0.98.1:40885)
I20260924 04:36:17.765877 7105 ts_manager.cc:194] Registered new tserver with Master: e4029b6d160547fdb897b6ee50f4b9ec (127.0.98.1:40885)
I20260924 04:36:17.854636 6963 heartbeater.cc:503] Master 127.0.98.62:33081 was elected leader, sending a full tablet report...
W20260924 04:36:17.866897 7476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:17.867126 7476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:17.867206 7476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:17.871106 7476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:17.871153 7476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:17.871248 7476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:17.876178 7476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46417
--tserver_master_addrs=127.0.98.62:33081,127.0.98.61:35263,127.0.98.60:37173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:17.877111 7476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:17.878069 7476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:17.887080 7484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:17.887081 7482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:17.887121 7481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:17.887440 7476 server_base.cc:1034] running on GCE node
I20260924 04:36:17.887784 7476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:17.888355 7476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:17.889559 7476 hybrid_clock.cc:648] HybridClock initialized: now 1790224577889548 us; error 29 us; skew 500 ppm
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:17.892102 7476 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:36:17.892899 7476 webserver.cc:460] Webserver started at http://127.0.98.2:46417/ using document root <none> and password file <none>
I20260924 04:36:17.893450 7476 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:17.893522 7476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:17.897197 7476 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 04:36:17.899299 7491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.900478 7476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:17.900621 7476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "b574eca291e34af69b610f262ea5dc21"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "e9e9fbe5ab0648a282c3dc5168a2ab75"
server_key_iv: "b42f51ad2b20d768371f30d57211fb3f"
server_key_version: "encryptionkey@0"
I20260924 04:36:17.901219 7476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:17.919961 7476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:17.922844 7476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:17.922986 7476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:17.923589 7476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:17.924424 7476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:17.924475 7476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.924542 7476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:17.924574 7476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:17.933564 7476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33661
I20260924 04:36:17.933559 7604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33661 every 8 connection(s)
I20260924 04:36:17.934899 7476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:17.939942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 7476
I20260924 04:36:17.940143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 6834
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:36:17 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224577, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:17.950045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40723
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33081
--webserver_port=43409
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33081,127.0.98.61:35263
--tserver_master_addrs=127.0.98.62:33081,127.0.98.61:35263,127.0.98.60:37173 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:36:17.953612 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.936762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39879 (local address 127.0.98.62:33081)
0924 04:36:17.936881 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.936886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.938409 (+ 1523us) server_negotiation.cc:407] Connection header received
0924 04:36:17.939058 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.939061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.939118 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.939216 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.941241 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.942153 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.944123 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.944560 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.944660 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.949598 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.949603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.949608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.949639 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.951222 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.951763 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.951767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.951769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.951867 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.952278 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.952281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.952283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.952489 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.952569 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.953332 (+ 763us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.953452 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:36:17.953612 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.938340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33019 (local address 127.0.98.60:37173)
0924 04:36:17.938462 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.938467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.938698 (+ 231us) server_negotiation.cc:407] Connection header received
0924 04:36:17.939196 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.939199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.939259 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.939333 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.941197 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.941998 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.943974 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.944323 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.944388 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.948831 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.948835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.948838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.948868 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.951055 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.951763 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.951767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.951769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.951867 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.952278 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.952281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.952283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.952489 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.952569 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.953332 (+ 763us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.953454 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:36:17.954350 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.936708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43885 (local address 127.0.98.61:35263)
0924 04:36:17.936837 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:17.936841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:17.937750 (+ 909us) server_negotiation.cc:407] Connection header received
0924 04:36:17.938898 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:17.938902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:17.938975 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:17.939081 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:17.941196 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.941998 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.943974 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.944304 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.944380 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.948856 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:17.948863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:17.948869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:17.948901 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:17.952095 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.952792 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.952798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.952803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.952887 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:17.953240 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:17.953247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:17.953251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:17.953500 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:17.953587 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:17.953979 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 04:36:17.954128 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:36:17.955518 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.937040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.98.2:43885)
0924 04:36:17.937595 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.937641 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.938692 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.939247 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.939255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.939847 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.941010 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.941031 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.942176 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.942179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.943842 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.943852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.944566 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.944569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.944830 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.945773 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.945786 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.948698 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.952414 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.952420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.952423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.952677 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.953009 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.953015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.953020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.953115 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.953646 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.953652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.953773 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.954503 (+ 730us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.954669 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":344,"spinlock_wait_cycles":3328,"thread_start_us":560,"threads_started":1}
I20260924 04:36:17.955781 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.937040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.2:39879)
0924 04:36:17.938309 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.938320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.938695 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.939247 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.939256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.939857 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.941056 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.941076 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.942326 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.942333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.943900 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.943914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.944846 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.944852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.945025 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.945693 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.945720 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.949403 (+ 3683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.951343 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.951346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.951347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.951648 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.951974 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.951977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.951979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.952167 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.952637 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.952644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.953014 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.953827 (+ 813us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.954292 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"mutex_wait_us":817,"spinlock_wait_cycles":105088,"thread_start_us":182,"threads_started":1}
I20260924 04:36:17.955523 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:17.938086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.98.2:33019)
0924 04:36:17.938582 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:17.938596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:17.938695 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:17.939463 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:17.939465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:17.939847 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:17.941049 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.941055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.942176 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.942179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.943842 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:17.943852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:17.944566 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:17.944569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:17.944833 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:17.945682 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:17.945711 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:17.948633 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:17.951179 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.951186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.951201 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.951648 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.951974 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:17.951977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:17.951979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:17.952167 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:17.952637 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:17.952644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:17.953014 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 04:36:17.954122 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0924 04:36:17.954342 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":1198,"spinlock_wait_cycles":32000,"thread_start_us":75,"threads_started":1}
I20260924 04:36:17.956980 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35263
I20260924 04:36:17.957026 7605 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37173
I20260924 04:36:17.957049 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33081
I20260924 04:36:17.957238 7607 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:17.957238 7606 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:17.957240 7605 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:17.957870 7605 heartbeater.cc:511] Master 127.0.98.60:37173 requested a full tablet report, sending...
I20260924 04:36:17.957872 7606 heartbeater.cc:511] Master 127.0.98.62:33081 requested a full tablet report, sending...
I20260924 04:36:17.957871 7607 heartbeater.cc:511] Master 127.0.98.61:35263 requested a full tablet report, sending...
I20260924 04:36:17.958995 7292 ts_manager.cc:194] Registered new tserver with Master: b574eca291e34af69b610f262ea5dc21 (127.0.98.2:33661)
I20260924 04:36:17.959033 6503 ts_manager.cc:194] Re-registered known tserver with Master: b574eca291e34af69b610f262ea5dc21 (127.0.98.2:33661)
I20260924 04:36:17.959477 7105 ts_manager.cc:194] Registered new tserver with Master: b574eca291e34af69b610f262ea5dc21 (127.0.98.2:33661)
I20260924 04:36:17.959858 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39879
W20260924 04:36:18.063019 7612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:18.063270 7612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:18.063313 7612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:18.067333 7612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:18.067396 7612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:18.067454 7612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:18.072531 7612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43409
--tserver_master_addrs=127.0.98.62:33081,127.0.98.61:35263,127.0.98.60:37173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:18.073581 7612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:18.074546 7612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:18.084030 7618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:18.084178 7620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:18.084299 7612 server_base.cc:1034] running on GCE node
W20260924 04:36:18.084444 7617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:18.084867 7612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:18.085608 7612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:18.086781 7612 hybrid_clock.cc:648] HybridClock initialized: now 1790224578086756 us; error 44 us; skew 500 ppm
Sep 24 04:36:18 dist-test-slave-jlnt krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224578, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:18.089645 7612 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:36:18.090499 7612 webserver.cc:460] Webserver started at http://127.0.98.3:43409/ using document root <none> and password file <none>
I20260924 04:36:18.091040 7612 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:18.091141 7612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:18.094955 7612 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:18.097187 7627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:18.098511 7612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:36:18.098663 7612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "9c83f256d94d401bb7d3b788ab2e248e"
format_stamp: "Formatted at 2026-09-24 04:36:16 on dist-test-slave-jlnt"
server_key: "eef705d382f0448bfe8bda29699ea98d"
server_key_iv: "433c0b3a3263aaa101c555f759a92a8a"
server_key_version: "encryptionkey@0"
I20260924 04:36:18.099177 7612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:18.126552 7612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:18.129949 7612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:18.130172 7612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:18.130764 7612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:18.131604 7612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:18.131675 7612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:18.131745 7612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:18.131788 7612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:18.140933 7612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40723
I20260924 04:36:18.140949 7740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40723 every 8 connection(s)
I20260924 04:36:18.141909 7612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:18.146927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 7612
Sep 24 04:36:18 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224578, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:36:18 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224578, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:36:18 dist-test-slave-jlnt krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224578, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:36:18.159976 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.143710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44311 (local address 127.0.98.62:33081)
0924 04:36:18.143842 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.143847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.144996 (+ 1149us) server_negotiation.cc:407] Connection header received
0924 04:36:18.145936 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.145940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.145993 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.146058 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.147456 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.148236 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.149966 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.150310 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.150403 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.155268 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.155272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.155274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.155302 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.157615 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.158215 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.158219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.158221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.158302 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.158653 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.158657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.158658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.158873 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.158953 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.159725 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.159829 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:36:18.160748 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.143757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36655 (local address 127.0.98.61:35263)
0924 04:36:18.143874 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.143878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.144768 (+ 890us) server_negotiation.cc:407] Connection header received
0924 04:36:18.145833 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.145837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.145895 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.146000 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.151473 (+ 5473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.152222 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.153398 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.153778 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.153850 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.156965 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.156968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.156971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.156994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.158755 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.159249 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.159255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.159259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.159337 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.159898 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.159902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.159904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.160130 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.160205 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.160515 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.160623 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:36:18.160771 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.145253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60415 (local address 127.0.98.60:37173)
0924 04:36:18.145386 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.145390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.145431 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:36:18.145864 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.145868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.145926 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.145977 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.147392 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.148160 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.150796 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.151114 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.151225 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.154864 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.154870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.154874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.154898 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.156609 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.158554 (+ 1945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.158560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.158563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.158657 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.159526 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.159530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.159532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.159762 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.159897 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.160516 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.160641 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:36:18.160858 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.144093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.3:44311)
0924 04:36:18.144914 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:18.144927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:18.145672 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:18.146174 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:18.146176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:18.146641 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:18.147244 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.147267 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.148394 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.148397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.149846 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.149857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.150545 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.150549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.150759 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.151609 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:18.151629 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:18.155123 (+ 3494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:18.157736 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:18.157741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:18.157756 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:18.158100 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:18.158395 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:18.158398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:18.158400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:18.158554 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:18.159000 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:18.159006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:18.159305 (+ 299us) client_negotiation.cc:769] Sending connection context
0924 04:36:18.159496 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 04:36:18.159823 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":147,"spinlock_wait_cycles":16768,"thread_start_us":155,"threads_started":1}
I20260924 04:36:18.161311 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.144093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.98.3:36655)
0924 04:36:18.144598 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:18.144645 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:18.145638 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:18.146168 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:18.146175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:18.146581 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:18.151338 (+ 4757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.151346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.152365 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.152371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.153261 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.153273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.154003 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.154009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.154102 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.154639 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:18.154657 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:18.156821 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:18.158861 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:18.158864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:18.158867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:18.159137 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:18.159433 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:18.159435 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:18.159437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:18.159528 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:18.160259 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:18.160261 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:18.160349 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 04:36:18.160943 (+ 594us) client_negotiation.cc:240] Negotiation successful
0924 04:36:18.161104 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":38,"spinlock_wait_cycles":4864,"thread_start_us":551,"threads_started":1}
I20260924 04:36:18.161526 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.145087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.98.3:60415)
0924 04:36:18.145359 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:18.145371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:18.145672 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:18.146168 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:18.146176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:18.146661 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:18.147255 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.147267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.148295 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.148301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.150672 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.150680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.151236 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.151239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.151341 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.153054 (+ 1713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:18.153068 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:18.154715 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:18.158153 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:18.158157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:18.158162 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:18.158417 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:18.159334 (+ 917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:18.159337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:18.159340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:18.159424 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:18.159941 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:18.159947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:18.160262 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 04:36:18.160353 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:36:18.160494 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1291,"spinlock_wait_cycles":11648,"thread_start_us":99,"threads_started":1}
I20260924 04:36:18.162058 7743 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35263
I20260924 04:36:18.162153 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33081
I20260924 04:36:18.162271 7742 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:18.162842 7742 heartbeater.cc:511] Master 127.0.98.62:33081 requested a full tablet report, sending...
I20260924 04:36:18.164263 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37173
I20260924 04:36:18.164340 7741 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:18.162268 7743 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:18.164500 7741 heartbeater.cc:511] Master 127.0.98.60:37173 requested a full tablet report, sending...
I20260924 04:36:18.164590 7743 heartbeater.cc:511] Master 127.0.98.61:35263 requested a full tablet report, sending...
I20260924 04:36:18.164870 6503 ts_manager.cc:194] Re-registered known tserver with Master: 9c83f256d94d401bb7d3b788ab2e248e (127.0.98.3:40723)
I20260924 04:36:18.165210 7105 ts_manager.cc:194] Registered new tserver with Master: 9c83f256d94d401bb7d3b788ab2e248e (127.0.98.3:40723)
I20260924 04:36:18.165581 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44311
I20260924 04:36:18.166940 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.148841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.0.98.62:33081)
0924 04:36:18.149054 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.149058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.149685 (+ 627us) server_negotiation.cc:407] Connection header received
0924 04:36:18.149876 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.149879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.149939 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.150026 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.151845 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.152836 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.154620 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.155032 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.155135 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.162778 (+ 7643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.162782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.162784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.162808 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.164863 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.165504 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.165509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.165512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.165593 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.165950 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.165953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.165955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.166161 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.166273 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.166608 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.166752 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
I20260924 04:36:18.167382 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.149324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.0.98.60:37173)
0924 04:36:18.154364 (+ 5040us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.154378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.154390 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:36:18.154426 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.154428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.154469 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.155992 (+ 1523us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.156885 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.157748 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.158842 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.159162 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.159249 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.163455 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.163460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.163464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.163492 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.165258 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.165747 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.165752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.165756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.165837 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.166127 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.166131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.166133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.166337 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.166437 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.167084 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.167224 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4972,"thread_start_us":76,"threads_started":1}
I20260924 04:36:18.167788 7292 ts_manager.cc:194] Registered new tserver with Master: 9c83f256d94d401bb7d3b788ab2e248e (127.0.98.3:40723)
I20260924 04:36:18.169064 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.148979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.0.98.61:35263)
0924 04:36:18.149470 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.149476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.149975 (+ 499us) server_negotiation.cc:407] Connection header received
0924 04:36:18.150146 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.150150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.150196 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.150263 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.151323 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.152110 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.153245 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.153583 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.153664 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.155498 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.155502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.155504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.155530 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.159234 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.163735 (+ 4501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.163738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.163741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.163828 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.168066 (+ 4238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.168069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.168071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.168267 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.168384 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.168795 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.168915 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":92,"threads_started":1}
I20260924 04:36:18.170184 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.163838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.0.98.60:37173)
0924 04:36:18.164447 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.164451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.166019 (+ 1568us) server_negotiation.cc:407] Connection header received
0924 04:36:18.166210 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.166213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.166270 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.166330 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:18.166658 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.167342 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.168752 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.169390 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.169517 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.169668 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.170020 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.170081 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542}
I20260924 04:36:18.170202 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.163560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.0.1:40610)
0924 04:36:18.165948 (+ 2388us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:18.165963 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:18.166097 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:18.166446 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:18.166449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:18.166465 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:18.166542 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.166548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.167480 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.167483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.168554 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.168563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.169800 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.169804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.169896 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.169912 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:36:18.170004 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:36:18.170085 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2324,"thread_start_us":67,"threads_started":1}
I20260924 04:36:18.170647 7302 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 04:36:18.171015 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27d96944a97f400ab082a7cf95b97dd7. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:18.171139 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:36:18.170941 6503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:36:18.173870 6503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:18.176343 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: true } } }
I20260924 04:36:18.176476 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:18.179577 7758 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: attempting to promote NON_VOTER 61fad00dd08f42808ded7d0901de51d1 to VOTER
I20260924 04:36:18.179905 7758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } }
I20260924 04:36:18.180867 7302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 LEARNER]: Refusing update from remote peer 27d96944a97f400ab082a7cf95b97dd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:36:18.180868 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Refusing update from remote peer 27d96944a97f400ab082a7cf95b97dd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:36:18.181246 7763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:36:18.181460 7759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:36:18.182732 7759 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 61fad00dd08f42808ded7d0901de51d1 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:18.183377 7302 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 61fad00dd08f42808ded7d0901de51d1 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:18.183598 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 61fad00dd08f42808ded7d0901de51d1 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:18.184197 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:18.184293 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:18.184404 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:18.184463 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:18.184540 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:18.184595 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:18.185191 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:18.185312 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:36:18.185214 7771 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27d96944a97f400ab082a7cf95b97dd7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:18.185534 7771 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:36:18.193410 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.188855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40885 (local address 127.0.0.1:39894)
0924 04:36:18.189005 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:18.189020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:18.189113 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:18.189959 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:18.189962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:18.189975 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:18.190077 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.190086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.191453 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.191460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.192215 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.192223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.193001 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.193004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.193096 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.193117 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:36:18.193219 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:36:18.193289 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":2}
I20260924 04:36:18.193464 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.188998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.98.1:40885)
0924 04:36:18.189477 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.189503 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.189525 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:36:18.189615 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.189621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.189842 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.189996 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:18.190171 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.191349 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.192342 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.192876 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.192993 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.193138 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.193247 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.193332 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":149,"threads_started":1}
I20260924 04:36:18.194609 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.189514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33661 (local address 127.0.0.1:40616)
0924 04:36:18.189800 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:18.189814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:18.189901 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:18.190414 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:18.190417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:18.190453 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:18.190546 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.190552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.191860 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.191863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.193016 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.193024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.194256 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.194259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.194344 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.194364 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:36:18.194432 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:36:18.194489 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20260924 04:36:18.195015 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.189658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.0.98.2:33661)
0924 04:36:18.190035 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.190053 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.190082 (+ 29us) server_negotiation.cc:407] Connection header received
0924 04:36:18.190148 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.190151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.190291 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.190442 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:18.190686 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.191739 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.193179 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.193970 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.194070 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.194195 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.194781 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.194876 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":90,"threads_started":1}
I20260924 04:36:18.195416 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.189027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40723 (local address 127.0.0.1:51070)
0924 04:36:18.189720 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:18.189733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:18.189810 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:18.191258 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:18.191261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:18.191277 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:18.191365 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.191371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.192697 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.192700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.193805 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.193816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.195007 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.195012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.195128 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.195152 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:36:18.195231 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:36:18.195305 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":247,"threads_started":1}
I20260924 04:36:18.195482 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.189352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.0.98.3:40723)
0924 04:36:18.190707 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.190721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.190743 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:36:18.190811 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.190816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.190992 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.191136 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:18.191478 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.192574 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.193942 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.194732 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.194830 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.194947 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.195263 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.195351 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1258,"thread_start_us":84,"threads_started":1}
I20260924 04:36:18.196620 7402 tablet_service.cc:1467] Processing CreateTablet for tablet e31b1e2ebc0c4f4e8e278065563e70fa (DEFAULT_TABLE table=test-table [id=91e2c8cb57944aeda85c3e513e2af998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:18.197656 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e31b1e2ebc0c4f4e8e278065563e70fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:18.197640 7539 tablet_service.cc:1467] Processing CreateTablet for tablet e31b1e2ebc0c4f4e8e278065563e70fa (DEFAULT_TABLE table=test-table [id=91e2c8cb57944aeda85c3e513e2af998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:18.197790 7675 tablet_service.cc:1467] Processing CreateTablet for tablet e31b1e2ebc0c4f4e8e278065563e70fa (DEFAULT_TABLE table=test-table [id=91e2c8cb57944aeda85c3e513e2af998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:18.198618 7539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e31b1e2ebc0c4f4e8e278065563e70fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:18.198639 7675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e31b1e2ebc0c4f4e8e278065563e70fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:18.204463 7777 tablet_bootstrap.cc:492] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: Bootstrap starting.
I20260924 04:36:18.206419 7778 tablet_bootstrap.cc:492] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: Bootstrap starting.
I20260924 04:36:18.207337 7779 tablet_bootstrap.cc:492] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: Bootstrap starting.
I20260924 04:36:18.207664 7777 tablet_bootstrap.cc:654] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:18.208436 7778 tablet_bootstrap.cc:654] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:18.208461 7777 log.cc:824] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:18.209077 7779 tablet_bootstrap.cc:654] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:18.209201 7778 log.cc:824] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:18.209908 7779 log.cc:824] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:18.210494 7777 tablet_bootstrap.cc:492] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: No bootstrap required, opened a new log
I20260924 04:36:18.210685 7777 ts_tablet_manager.cc:1397] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:36:18.211190 7778 tablet_bootstrap.cc:492] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: No bootstrap required, opened a new log
I20260924 04:36:18.211359 7778 ts_tablet_manager.cc:1397] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:36:18.211622 7779 tablet_bootstrap.cc:492] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: No bootstrap required, opened a new log
I20260924 04:36:18.211805 7779 ts_tablet_manager.cc:1397] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 04:36:18.215278 7779 raft_consensus.cc:348] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:18.215526 7779 raft_consensus.cc:374] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:18.215574 7779 raft_consensus.cc:729] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b574eca291e34af69b610f262ea5dc21, State: Initialized, Role: FOLLOWER
I20260924 04:36:18.215794 7777 raft_consensus.cc:348] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:18.216034 7777 raft_consensus.cc:374] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:18.215991 7779 consensus_queue.cc:260] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:18.216099 7777 raft_consensus.cc:729] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4029b6d160547fdb897b6ee50f4b9ec, State: Initialized, Role: FOLLOWER
I20260924 04:36:18.216254 7778 raft_consensus.cc:348] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:18.216475 7778 raft_consensus.cc:374] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:18.216531 7778 raft_consensus.cc:729] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c83f256d94d401bb7d3b788ab2e248e, State: Initialized, Role: FOLLOWER
I20260924 04:36:18.216487 7777 consensus_queue.cc:260] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:18.216936 7779 ts_tablet_manager.cc:1428] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:36:18.216920 7778 consensus_queue.cc:260] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:18.217432 7777 ts_tablet_manager.cc:1428] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 04:36:18.217486 7469 heartbeater.cc:503] Master 127.0.98.62:33081 was elected leader, sending a full tablet report...
I20260924 04:36:18.217939 7778 ts_tablet_manager.cc:1428] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260924 04:36:18.219051 7742 heartbeater.cc:503] Master 127.0.98.62:33081 was elected leader, sending a full tablet report...
I20260924 04:36:18.219353 7606 heartbeater.cc:503] Master 127.0.98.62:33081 was elected leader, sending a full tablet report...
W20260924 04:36:18.242362 7471 tablet.cc:2369] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:36:18.282799 7783 raft_consensus.cc:482] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:18.283031 7783 raft_consensus.cc:504] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:18.284211 7783 leader_election.cc:288] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4029b6d160547fdb897b6ee50f4b9ec (127.0.98.1:40885), 9c83f256d94d401bb7d3b788ab2e248e (127.0.98.3:40723)
I20260924 04:36:18.288138 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.284509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35549 (local address 127.0.98.1:40885)
0924 04:36:18.284618 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.284623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.284674 (+ 51us) server_negotiation.cc:407] Connection header received
0924 04:36:18.284837 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.284841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.284894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.284971 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:18.285316 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.286061 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.287182 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.287579 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.287696 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.287758 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.287938 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.288010 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:36:18.288139 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.284489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40723 (local address 127.0.98.2:43483)
0924 04:36:18.284877 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:18.284889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:18.284962 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:18.285241 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:18.285243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:18.285273 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:18.285379 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.285385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.286227 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.286232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.287063 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.287072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.287730 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.287733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.287845 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.287862 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:36:18.287957 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:36:18.288020 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":46}
I20260924 04:36:18.288160 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.284547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43483 (local address 127.0.98.3:40723)
0924 04:36:18.284690 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.284693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.284963 (+ 270us) server_negotiation.cc:407] Connection header received
0924 04:36:18.285049 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.285053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.285108 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.285218 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:18.285492 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.286107 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.287211 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.287588 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.287706 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.287765 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.287990 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.288062 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:36:18.288141 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.284398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40885 (local address 127.0.98.2:35549)
0924 04:36:18.284598 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:18.284614 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:18.284720 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:18.285006 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:18.285010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:18.285034 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:18.285201 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.285211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.286171 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.286177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.287040 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:18.287048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.287693 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.287697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:18.287811 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.287836 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:36:18.287896 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:36:18.287983 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":11}
I20260924 04:36:18.288839 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e31b1e2ebc0c4f4e8e278065563e70fa" candidate_uuid: "b574eca291e34af69b610f262ea5dc21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" is_pre_election: true
I20260924 04:36:18.288946 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e31b1e2ebc0c4f4e8e278065563e70fa" candidate_uuid: "b574eca291e34af69b610f262ea5dc21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c83f256d94d401bb7d3b788ab2e248e" is_pre_election: true
I20260924 04:36:18.289213 7422 raft_consensus.cc:2454] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b574eca291e34af69b610f262ea5dc21 in term 0.
I20260924 04:36:18.289258 7695 raft_consensus.cc:2454] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b574eca291e34af69b610f262ea5dc21 in term 0.
I20260924 04:36:18.289698 7495 leader_election.cc:302] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b574eca291e34af69b610f262ea5dc21, e4029b6d160547fdb897b6ee50f4b9ec; no voters:
I20260924 04:36:18.290009 7783 raft_consensus.cc:2781] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:18.290099 7783 raft_consensus.cc:482] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:18.290143 7783 raft_consensus.cc:3037] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:18.291177 7783 raft_consensus.cc:504] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:18.291491 7783 leader_election.cc:288] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [CANDIDATE]: Term 1 election: Requested vote from peers e4029b6d160547fdb897b6ee50f4b9ec (127.0.98.1:40885), 9c83f256d94d401bb7d3b788ab2e248e (127.0.98.3:40723)
I20260924 04:36:18.291803 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e31b1e2ebc0c4f4e8e278065563e70fa" candidate_uuid: "b574eca291e34af69b610f262ea5dc21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4029b6d160547fdb897b6ee50f4b9ec"
I20260924 04:36:18.291803 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e31b1e2ebc0c4f4e8e278065563e70fa" candidate_uuid: "b574eca291e34af69b610f262ea5dc21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c83f256d94d401bb7d3b788ab2e248e"
I20260924 04:36:18.291935 7695 raft_consensus.cc:3037] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:18.291925 7422 raft_consensus.cc:3037] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:18.293123 7422 raft_consensus.cc:2454] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b574eca291e34af69b610f262ea5dc21 in term 1.
I20260924 04:36:18.293123 7695 raft_consensus.cc:2454] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b574eca291e34af69b610f262ea5dc21 in term 1.
I20260924 04:36:18.293412 7494 leader_election.cc:302] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c83f256d94d401bb7d3b788ab2e248e, b574eca291e34af69b610f262ea5dc21; no voters:
I20260924 04:36:18.293565 7783 raft_consensus.cc:2781] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:18.293784 7783 raft_consensus.cc:686] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 1 LEADER]: Becoming Leader. State: Replica: b574eca291e34af69b610f262ea5dc21, State: Running, Role: LEADER
I20260924 04:36:18.294121 7783 consensus_queue.cc:237] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:18.297622 6502 catalog_manager.cc:5495] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 reported cstate change: term changed from 0 to 1, leader changed from <none> to b574eca291e34af69b610f262ea5dc21 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b574eca291e34af69b610f262ea5dc21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } health_report { overall_health: HEALTHY } } }
I20260924 04:36:18.317327 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.313859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.0.98.2:33661)
0924 04:36:18.313979 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.313984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.313997 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:18.314165 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.314167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.314218 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.314317 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:18.314573 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.315072 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.315912 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.316143 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.316252 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.316542 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:18.316935 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:18.317075 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.317110 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.317194 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:36:18.322929 7422 raft_consensus.cc:1261] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [term 1 FOLLOWER]: Refusing update from remote peer b574eca291e34af69b610f262ea5dc21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:18.322929 7695 raft_consensus.cc:1261] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [term 1 FOLLOWER]: Refusing update from remote peer b574eca291e34af69b610f262ea5dc21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:18.323704 7783 consensus_queue.cc:1035] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:18.323875 7786 consensus_queue.cc:1035] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:18.333966 7789 mvcc.cc:205] Tried to move back new op lower bound from 7332759872803880960 to 7332759872695373824. Current Snapshot: MvccSnapshot[applied={T|T < 7332759872803880960 or (T in {7332759872803880960})}]
I20260924 04:36:18.334148 7791 mvcc.cc:205] Tried to move back new op lower bound from 7332759872803880960 to 7332759872695373824. Current Snapshot: MvccSnapshot[applied={T|T < 7332759872803880960}]
I20260924 04:36:18.340143 6504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:18.340270 6504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:18.344035 6504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27d96944a97f400ab082a7cf95b97dd7: Sending DeleteTablet for 3 replicas of tablet e31b1e2ebc0c4f4e8e278065563e70fa
I20260924 04:36:18.344807 7675 tablet_service.cc:1514] Processing DeleteTablet for tablet e31b1e2ebc0c4f4e8e278065563e70fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51070
I20260924 04:36:18.344831 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet e31b1e2ebc0c4f4e8e278065563e70fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39894
I20260924 04:36:18.344882 7539 tablet_service.cc:1514] Processing DeleteTablet for tablet e31b1e2ebc0c4f4e8e278065563e70fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40616
I20260924 04:36:18.348177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 6472
I20260924 04:36:18.353363 7800 tablet_replica.cc:330] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: stopping tablet replica
I20260924 04:36:18.353740 7800 raft_consensus.cc:2229] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:18.354092 7800 raft_consensus.cc:2258] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:18.355659 7800 ts_tablet_manager.cc:1905] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:18.356727 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.347500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.0.98.61:35263)
0924 04:36:18.347622 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.347627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.347640 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:18.347675 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.347678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.347720 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.347794 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.349595 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.350122 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.350959 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.351213 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.351321 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.352965 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.352968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.352971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.352994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.354939 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.355413 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.355419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.355424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.355519 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.355841 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.355844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.355846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.356052 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.356201 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.356465 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.356581 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:36:18.358505 7800 ts_tablet_manager.cc:1918] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:18.358603 7800 log.cc:1196] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/e31b1e2ebc0c4f4e8e278065563e70fa
I20260924 04:36:18.358947 7800 ts_tablet_manager.cc:1939] T e31b1e2ebc0c4f4e8e278065563e70fa P e4029b6d160547fdb897b6ee50f4b9ec: Deleting consensus metadata
W20260924 04:36:18.360046 7357 connection.cc:441] server connection from 127.0.0.1:39894 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39894 (request call id 1) could send its response
I20260924 04:36:18.360142 7799 tablet_replica.cc:330] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: stopping tablet replica
I20260924 04:36:18.360435 7799 raft_consensus.cc:2229] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:18.360733 7799 raft_consensus.cc:2258] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:18.361052 7798 tablet_replica.cc:330] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: stopping tablet replica
I20260924 04:36:18.361336 7798 raft_consensus.cc:2229] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:18.361644 7798 raft_consensus.cc:2258] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:18.363193 7799 ts_tablet_manager.cc:1905] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:18.363507 7798 ts_tablet_manager.cc:1905] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:18.365767 7799 ts_tablet_manager.cc:1918] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:18.365844 7799 log.cc:1196] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/e31b1e2ebc0c4f4e8e278065563e70fa
I20260924 04:36:18.366148 7799 ts_tablet_manager.cc:1939] T e31b1e2ebc0c4f4e8e278065563e70fa P 9c83f256d94d401bb7d3b788ab2e248e: Deleting consensus metadata
I20260924 04:36:18.366350 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.347192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.98.60:37173)
0924 04:36:18.347305 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.347308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.356896 (+ 9588us) server_negotiation.cc:407] Connection header received
0924 04:36:18.356939 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.356945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.356999 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.357080 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.358178 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.358894 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.359790 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.360112 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.360212 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.362634 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.362638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.362640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.362666 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.364328 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.364795 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.364799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.364800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.364882 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.365235 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.365241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.365246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.365432 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.365540 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.366100 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.366203 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:36:18.367116 7631 connection.cc:441] server connection from 127.0.0.1:51070 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51070 (request call id 1) could send its response
I20260924 04:36:18.370218 7798 ts_tablet_manager.cc:1918] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:18.370289 7798 log.cc:1196] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/e31b1e2ebc0c4f4e8e278065563e70fa
I20260924 04:36:18.370535 7798 ts_tablet_manager.cc:1939] T e31b1e2ebc0c4f4e8e278065563e70fa P b574eca291e34af69b610f262ea5dc21: Deleting consensus metadata
W20260924 04:36:18.371524 7492 connection.cc:441] server connection from 127.0.0.1:40616 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40616 (request call id 1) could send its response
I20260924 04:36:18.375901 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.366673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.0.98.61:35263)
0924 04:36:18.366767 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.366771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.367077 (+ 306us) server_negotiation.cc:407] Connection header received
0924 04:36:18.367231 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.367234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.367273 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.367315 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.368197 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.368959 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.370102 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.370433 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.370550 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.372124 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.372127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.372130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.372153 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.374040 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.374521 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.374524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.374525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.374617 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.374861 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.374866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.374867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.375092 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.375200 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.375634 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.375756 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:36:18.376221 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:18.361113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.0.98.60:37173)
0924 04:36:18.361234 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:18.361237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:18.361248 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:36:18.361283 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:18.361288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:18.361332 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:18.361413 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:18.364790 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.365803 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.367739 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:18.368119 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:18.368191 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:18.372678 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:18.372684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:18.372689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:18.372717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:18.374448 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.374934 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.374938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.374940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.375008 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:18.375303 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:18.375306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:18.375307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:18.375564 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:18.375681 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:18.375935 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 04:36:18.376060 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:36:19.335801 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.335049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.2:43785)
0924 04:36:19.335570 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.335685 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":79,"threads_started":1}
I20260924 04:36:19.335834 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.335256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.3:57081)
0924 04:36:19.335643 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.335735 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":62,"threads_started":1}
W20260924 04:36:19.336103 7606 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33081 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
W20260924 04:36:19.336421 7742 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33081 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
I20260924 04:36:19.339579 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.339221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.1:49325)
0924 04:36:19.339425 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.339501 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":56,"threads_started":1}
W20260924 04:36:19.339852 7469 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33081 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
I20260924 04:36:19.844830 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27d96944a97f400ab082a7cf95b97dd7)
I20260924 04:36:19.845077 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } }
I20260924 04:36:19.846145 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27d96944a97f400ab082a7cf95b97dd7 (127.0.98.62:33081), a83a227cb46d4afb8fc2faf1d7907a98 (127.0.98.61:35263)
I20260924 04:36:19.846505 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27d96944a97f400ab082a7cf95b97dd7)
I20260924 04:36:19.846658 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.846222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:34104)
0924 04:36:19.846477 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.846560 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
W20260924 04:36:19.846827 7273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
I20260924 04:36:19.846680 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } }
I20260924 04:36:19.847746 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.847530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:34120)
0924 04:36:19.847632 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.847682 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 04:36:19.847921 7273 leader_election.cc:334] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27d96944a97f400ab082a7cf95b97dd7 (127.0.98.62:33081): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
I20260924 04:36:19.848254 7813 leader_election.cc:288] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27d96944a97f400ab082a7cf95b97dd7 (127.0.98.62:33081), 61fad00dd08f42808ded7d0901de51d1 (127.0.98.60:37173)
I20260924 04:36:19.848635 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.848280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:34132)
0924 04:36:19.848479 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.848553 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":50,"threads_started":1}
W20260924 04:36:19.848856 7089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
I20260924 04:36:19.849745 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.849524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:34148)
0924 04:36:19.849637 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.849691 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:36:19.849856 7089 leader_election.cc:334] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27d96944a97f400ab082a7cf95b97dd7 (127.0.98.62:33081): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
I20260924 04:36:19.851212 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.846374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35263 (local address 127.0.0.1:48040)
0924 04:36:19.846639 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.846655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:19.846763 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:19.847164 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:19.847167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:19.847185 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:19.847400 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.847413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.848479 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.848484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.850019 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.850029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.850901 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.850904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.850971 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.850991 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:36:19.851061 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:36:19.851115 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":51,"threads_started":1}
I20260924 04:36:19.851279 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.846558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.0.98.61:35263)
0924 04:36:19.846894 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:19.846900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:19.846916 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:36:19.846963 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:19.846966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:19.847018 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:19.847125 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:19.847510 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.848386 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.850151 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.850783 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.850885 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.850942 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:19.851103 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 04:36:19.851171 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":264,"thread_start_us":85,"threads_started":1}
I20260924 04:36:19.851799 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61fad00dd08f42808ded7d0901de51d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" is_pre_election: true
I20260924 04:36:19.851991 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61fad00dd08f42808ded7d0901de51d1 in term 1.
I20260924 04:36:19.852304 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61fad00dd08f42808ded7d0901de51d1, a83a227cb46d4afb8fc2faf1d7907a98; no voters: 27d96944a97f400ab082a7cf95b97dd7
I20260924 04:36:19.852545 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.848618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37173 (local address 127.0.0.1:51528)
0924 04:36:19.848768 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.848781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:19.848863 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:19.849250 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:19.849252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:19.849270 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:19.849351 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.849359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.850266 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.850269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.851406 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.851414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.852175 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.852178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.852271 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.852293 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:36:19.852369 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:36:19.852436 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260924 04:36:19.852602 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.848701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.0.98.60:37173)
0924 04:36:19.849004 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:19.849012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:19.849027 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:19.849072 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:19.849078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:19.849155 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:19.849240 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:19.849462 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.850174 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.851526 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.852045 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.852176 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.852239 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:19.852418 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 04:36:19.852494 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":82,"threads_started":1}
I20260924 04:36:19.852713 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:36:19.852778 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27d96944a97f400ab082a7cf95b97dd7)
I20260924 04:36:19.852823 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:36:19.853264 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "61fad00dd08f42808ded7d0901de51d1" is_pre_election: true
I20260924 04:36:19.853940 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } }
I20260924 04:36:19.854178 7302 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a83a227cb46d4afb8fc2faf1d7907a98 in current term 2: Already voted for candidate 61fad00dd08f42808ded7d0901de51d1 in this term.
I20260924 04:36:19.854314 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [CANDIDATE]: Term 2 election: Requested vote from peers 27d96944a97f400ab082a7cf95b97dd7 (127.0.98.62:33081), a83a227cb46d4afb8fc2faf1d7907a98 (127.0.98.61:35263)
I20260924 04:36:19.854566 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.854384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:34154)
0924 04:36:19.854467 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.854513 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 04:36:19.854583 7089 leader_election.cc:302] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a83a227cb46d4afb8fc2faf1d7907a98; no voters: 27d96944a97f400ab082a7cf95b97dd7, 61fad00dd08f42808ded7d0901de51d1
I20260924 04:36:19.854801 7813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:36:19.855120 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61fad00dd08f42808ded7d0901de51d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a83a227cb46d4afb8fc2faf1d7907a98"
I20260924 04:36:19.855291 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.855002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:34170)
0924 04:36:19.855159 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.855236 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 04:36:19.855379 7273 leader_election.cc:334] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 27d96944a97f400ab082a7cf95b97dd7 (127.0.98.62:33081): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
I20260924 04:36:19.855901 7813 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:36:19.856814 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61fad00dd08f42808ded7d0901de51d1 in term 2.
I20260924 04:36:19.857074 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61fad00dd08f42808ded7d0901de51d1, a83a227cb46d4afb8fc2faf1d7907a98; no voters: 27d96944a97f400ab082a7cf95b97dd7
I20260924 04:36:19.857337 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:36:19.857553 7811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [term 2 LEADER]: Becoming Leader. State: Replica: 61fad00dd08f42808ded7d0901de51d1, State: Running, Role: LEADER
I20260924 04:36:19.857792 7811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } }
I20260924 04:36:19.859078 7821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61fad00dd08f42808ded7d0901de51d1. Latest consensus state: current_term: 2 leader_uuid: "61fad00dd08f42808ded7d0901de51d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:19.859179 7821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:19.859460 7823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:19.861421 7823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:19.861769 7823 catalog_manager.cc:1259] Loaded cluster ID: a699fc3a67f940ae8e60c856aea58cb6
I20260924 04:36:19.861814 7823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:19.862277 7823 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:19.862622 7823 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Loaded TSK: 0
I20260924 04:36:19.863220 7823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:36:19.889001 7292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:36:19.892006 7292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:19.894999 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.894739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:34190)
0924 04:36:19.894866 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.894923 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:36:19.895004 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [term 2 FOLLOWER]: Refusing update from remote peer 61fad00dd08f42808ded7d0901de51d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 04:36:19.895473 7811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 04:36:19.895701 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.895470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.0.1:34192)
0924 04:36:19.895567 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.895627 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:36:19.895821 7273 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 -> Peer 27d96944a97f400ab082a7cf95b97dd7 (127.0.98.62:33081): Couldn't send request to peer 27d96944a97f400ab082a7cf95b97dd7. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:36:19.897086 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61fad00dd08f42808ded7d0901de51d1. Latest consensus state: current_term: 2 leader_uuid: "61fad00dd08f42808ded7d0901de51d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:19.897249 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:36:19.898352 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "61fad00dd08f42808ded7d0901de51d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:19.898489 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:19.898641 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "61fad00dd08f42808ded7d0901de51d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "27d96944a97f400ab082a7cf95b97dd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33081 } } peers { permanent_uuid: "a83a227cb46d4afb8fc2faf1d7907a98" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35263 } attrs { promote: false } } peers { permanent_uuid: "61fad00dd08f42808ded7d0901de51d1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37173 } attrs { promote: false } } }
I20260924 04:36:19.898757 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83a227cb46d4afb8fc2faf1d7907a98 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:36:19.909469 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.904799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40885 (local address 127.0.0.1:43960)
0924 04:36:19.904911 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.904925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:19.905000 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:19.905846 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:19.905850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:19.905865 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:19.905973 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.905982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.907002 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.907008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.908158 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.908167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.909071 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.909075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.909189 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.909226 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 04:36:19.909277 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:36:19.909344 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":5}
I20260924 04:36:19.909778 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.905358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40723 (local address 127.0.0.1:53304)
0924 04:36:19.905441 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.905454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:19.905528 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:19.906012 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:19.906015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:19.906044 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:19.906162 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.906168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.907273 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.907276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.908415 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.908423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.909285 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.909289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.909435 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.909508 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 04:36:19.909579 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:36:19.909649 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":48,"spinlock_wait_cycles":1792}
I20260924 04:36:19.909919 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.905467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.0.98.3:40723)
0924 04:36:19.905739 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:19.905744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:19.905760 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:36:19.905815 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:19.905821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:19.905878 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:19.906266 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:19.906372 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.907165 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.908532 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.909175 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.909280 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.909337 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:19.909709 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 04:36:19.909776 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
I20260924 04:36:19.910121 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.905037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.0.98.1:40885)
0924 04:36:19.905327 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:19.905331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:19.905346 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:19.905659 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:19.905663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:19.905723 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:19.906085 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:19.906247 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.906910 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.908281 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.908924 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.909771 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.909854 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:19.909900 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:36:19.909970 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260924 04:36:19.910254 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.905677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.0.98.2:33661)
0924 04:36:19.906253 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:19.906257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:19.906269 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:36:19.906314 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:19.906316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:19.906364 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:19.906447 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:19.906679 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.907381 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.908389 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.909004 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.909116 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.909212 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:19.910020 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 04:36:19.910107 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":87,"threads_started":1}
I20260924 04:36:19.910471 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.905574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33661 (local address 127.0.0.1:55046)
0924 04:36:19.905946 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.905961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:19.906053 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:19.906473 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:19.906476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:19.906493 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:19.906568 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.906573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.907491 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.907495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.908268 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:19.908278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:19.909285 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:19.909289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:19.909433 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:19.909510 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 04:36:19.909579 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:36:19.909649 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":61,"spinlock_wait_cycles":4864,"thread_start_us":86,"threads_started":1}
I20260924 04:36:19.910425 7675 tablet_service.cc:1467] Processing CreateTablet for tablet 765cc639701a4943ad0e86458b0f5c59 (DEFAULT_TABLE table=test-table [id=c24d1daf4a8940cd970cd65d84ba9a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:19.910586 7402 tablet_service.cc:1467] Processing CreateTablet for tablet 765cc639701a4943ad0e86458b0f5c59 (DEFAULT_TABLE table=test-table [id=c24d1daf4a8940cd970cd65d84ba9a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:19.910709 7675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765cc639701a4943ad0e86458b0f5c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:19.910813 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765cc639701a4943ad0e86458b0f5c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:19.910974 7539 tablet_service.cc:1467] Processing CreateTablet for tablet 765cc639701a4943ad0e86458b0f5c59 (DEFAULT_TABLE table=test-table [id=c24d1daf4a8940cd970cd65d84ba9a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:19.911224 7539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765cc639701a4943ad0e86458b0f5c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:19.913553 7835 tablet_bootstrap.cc:492] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: Bootstrap starting.
I20260924 04:36:19.913717 7833 tablet_bootstrap.cc:492] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec: Bootstrap starting.
I20260924 04:36:19.914469 7835 tablet_bootstrap.cc:654] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:19.914469 7833 tablet_bootstrap.cc:654] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:19.915167 7834 tablet_bootstrap.cc:492] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: Bootstrap starting.
I20260924 04:36:19.915758 7833 tablet_bootstrap.cc:492] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec: No bootstrap required, opened a new log
I20260924 04:36:19.915858 7833 ts_tablet_manager.cc:1397] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:19.916044 7835 tablet_bootstrap.cc:492] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: No bootstrap required, opened a new log
I20260924 04:36:19.916178 7835 ts_tablet_manager.cc:1397] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:36:19.916213 7834 tablet_bootstrap.cc:654] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:19.916368 7833 raft_consensus.cc:348] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:19.916481 7833 raft_consensus.cc:374] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:19.916515 7833 raft_consensus.cc:729] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4029b6d160547fdb897b6ee50f4b9ec, State: Initialized, Role: FOLLOWER
I20260924 04:36:19.916587 7835 raft_consensus.cc:348] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:19.916667 7833 consensus_queue.cc:260] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:19.916728 7835 raft_consensus.cc:374] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:19.916764 7835 raft_consensus.cc:729] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b574eca291e34af69b610f262ea5dc21, State: Initialized, Role: FOLLOWER
I20260924 04:36:19.916889 7835 consensus_queue.cc:260] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:19.917012 7833 ts_tablet_manager.cc:1428] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:36:19.917356 7834 tablet_bootstrap.cc:492] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: No bootstrap required, opened a new log
I20260924 04:36:19.917423 7835 ts_tablet_manager.cc:1428] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:19.917438 7834 ts_tablet_manager.cc:1397] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:19.917846 7834 raft_consensus.cc:348] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:19.917961 7834 raft_consensus.cc:374] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:19.917997 7834 raft_consensus.cc:729] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c83f256d94d401bb7d3b788ab2e248e, State: Initialized, Role: FOLLOWER
I20260924 04:36:19.918121 7834 consensus_queue.cc:260] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:19.918406 7834 ts_tablet_manager.cc:1428] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:19.920593 7468 heartbeater.cc:503] Master 127.0.98.60:37173 was elected leader, sending a full tablet report...
I20260924 04:36:19.920636 7605 heartbeater.cc:503] Master 127.0.98.60:37173 was elected leader, sending a full tablet report...
I20260924 04:36:19.920840 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.919125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.3:42427)
0924 04:36:19.920684 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.920742 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1489,"thread_start_us":90,"threads_started":1}
I20260924 04:36:19.921204 7741 heartbeater.cc:503] Master 127.0.98.60:37173 was elected leader, sending a full tablet report...
I20260924 04:36:19.921890 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.917477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.1:58571)
0924 04:36:19.921729 (+ 4252us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.921788 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4184,"thread_start_us":85,"threads_started":1}
I20260924 04:36:19.922243 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.919084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.2:40649)
0924 04:36:19.922090 (+ 3006us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.922146 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2946,"thread_start_us":98,"threads_started":1}
I20260924 04:36:19.922868 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.921827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.3:58689)
0924 04:36:19.922757 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.922809 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":865}
I20260924 04:36:19.922879 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.922624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.2:45113)
0924 04:36:19.922784 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.922824 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":34}
I20260924 04:36:19.923379 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:19.923139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.1:47081)
0924 04:36:19.923254 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:19.923301 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 04:36:19.937443 7608 tablet.cc:2369] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:36:20.133725 7840 raft_consensus.cc:482] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:20.133869 7840 raft_consensus.cc:504] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:20.134445 7840 leader_election.cc:288] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4029b6d160547fdb897b6ee50f4b9ec (127.0.98.1:40885), 9c83f256d94d401bb7d3b788ab2e248e (127.0.98.3:40723)
I20260924 04:36:20.135008 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765cc639701a4943ad0e86458b0f5c59" candidate_uuid: "b574eca291e34af69b610f262ea5dc21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" is_pre_election: true
I20260924 04:36:20.135142 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765cc639701a4943ad0e86458b0f5c59" candidate_uuid: "b574eca291e34af69b610f262ea5dc21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c83f256d94d401bb7d3b788ab2e248e" is_pre_election: true
I20260924 04:36:20.135228 7422 raft_consensus.cc:2454] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b574eca291e34af69b610f262ea5dc21 in term 0.
I20260924 04:36:20.135296 7695 raft_consensus.cc:2454] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b574eca291e34af69b610f262ea5dc21 in term 0.
I20260924 04:36:20.135547 7495 leader_election.cc:302] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b574eca291e34af69b610f262ea5dc21, e4029b6d160547fdb897b6ee50f4b9ec; no voters:
I20260924 04:36:20.135758 7840 raft_consensus.cc:2781] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:20.135830 7840 raft_consensus.cc:482] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:20.135883 7840 raft_consensus.cc:3037] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:20.136942 7840 raft_consensus.cc:504] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:20.137328 7840 leader_election.cc:288] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [CANDIDATE]: Term 1 election: Requested vote from peers e4029b6d160547fdb897b6ee50f4b9ec (127.0.98.1:40885), 9c83f256d94d401bb7d3b788ab2e248e (127.0.98.3:40723)
I20260924 04:36:20.137656 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765cc639701a4943ad0e86458b0f5c59" candidate_uuid: "b574eca291e34af69b610f262ea5dc21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c83f256d94d401bb7d3b788ab2e248e"
I20260924 04:36:20.137722 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765cc639701a4943ad0e86458b0f5c59" candidate_uuid: "b574eca291e34af69b610f262ea5dc21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4029b6d160547fdb897b6ee50f4b9ec"
I20260924 04:36:20.137787 7695 raft_consensus.cc:3037] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:20.137841 7422 raft_consensus.cc:3037] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:20.138729 7422 raft_consensus.cc:2454] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b574eca291e34af69b610f262ea5dc21 in term 1.
I20260924 04:36:20.138752 7695 raft_consensus.cc:2454] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b574eca291e34af69b610f262ea5dc21 in term 1.
I20260924 04:36:20.139008 7495 leader_election.cc:302] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b574eca291e34af69b610f262ea5dc21, e4029b6d160547fdb897b6ee50f4b9ec; no voters:
I20260924 04:36:20.139186 7840 raft_consensus.cc:2781] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:20.139423 7840 raft_consensus.cc:686] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 1 LEADER]: Becoming Leader. State: Replica: b574eca291e34af69b610f262ea5dc21, State: Running, Role: LEADER
I20260924 04:36:20.139580 7840 consensus_queue.cc:237] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } }
I20260924 04:36:20.140098 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:20.139881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.2:52159)
0924 04:36:20.139975 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:20.140035 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 04:36:20.144438 7744 tablet.cc:2369] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:36:20.145195 7291 catalog_manager.cc:5495] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 reported cstate change: term changed from 0 to 1, leader changed from <none> to b574eca291e34af69b610f262ea5dc21 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b574eca291e34af69b610f262ea5dc21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b574eca291e34af69b610f262ea5dc21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33661 } health_report { overall_health: HEALTHY } } }
I20260924 04:36:20.199573 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:20.196556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.0.98.2:33661)
0924 04:36:20.196722 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:20.196726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:20.196738 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:36:20.196782 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:20.196784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:20.196828 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:20.196895 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:20.197104 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:20.197782 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:20.198719 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:20.198931 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:20.199028 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:20.199226 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:20.199290 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:20.199343 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:20.199421 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 04:36:20.199464 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 04:36:20.201642 7422 raft_consensus.cc:1261] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [term 1 FOLLOWER]: Refusing update from remote peer b574eca291e34af69b610f262ea5dc21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:20.201642 7695 raft_consensus.cc:1261] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [term 1 FOLLOWER]: Refusing update from remote peer b574eca291e34af69b610f262ea5dc21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:20.202139 7840 consensus_queue.cc:1035] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c83f256d94d401bb7d3b788ab2e248e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:20.202382 7845 consensus_queue.cc:1035] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4029b6d160547fdb897b6ee50f4b9ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:20.203873 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:20.203574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.3:48691)
0924 04:36:20.203738 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:20.203799 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 04:36:20.205094 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:20.204614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.1:51163)
0924 04:36:20.204954 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:20.205023 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262}
I20260924 04:36:20.206862 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:20.206533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.2:50257)
0924 04:36:20.206744 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:20.206797 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260924 04:36:20.207541 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:20.207331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.3:44527)
0924 04:36:20.207421 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:20.207470 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 04:36:20.209612 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:20.209406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.2:53413)
0924 04:36:20.209516 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:20.209564 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 04:36:20.212049 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:20.211798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33081 (local address 127.0.98.1:41025)
0924 04:36:20.211906 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:20.211955 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:36:20.212226 7290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:20.212350 7290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:20.215947 7290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61fad00dd08f42808ded7d0901de51d1: Sending DeleteTablet for 3 replicas of tablet 765cc639701a4943ad0e86458b0f5c59
I20260924 04:36:20.216738 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet 765cc639701a4943ad0e86458b0f5c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:20 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43960
I20260924 04:36:20.216837 7675 tablet_service.cc:1514] Processing DeleteTablet for tablet 765cc639701a4943ad0e86458b0f5c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:20 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53304
I20260924 04:36:20.216917 7539 tablet_service.cc:1514] Processing DeleteTablet for tablet 765cc639701a4943ad0e86458b0f5c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:20 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55046
I20260924 04:36:20.217196 7858 tablet_replica.cc:330] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: stopping tablet replica
I20260924 04:36:20.217257 7857 tablet_replica.cc:330] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec: stopping tablet replica
I20260924 04:36:20.217365 7857 raft_consensus.cc:2229] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:20.217430 7859 tablet_replica.cc:330] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: stopping tablet replica
I20260924 04:36:20.217489 7857 raft_consensus.cc:2258] T 765cc639701a4943ad0e86458b0f5c59 P e4029b6d160547fdb897b6ee50f4b9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:20.217578 7859 raft_consensus.cc:2229] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:20.217794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 7249
I20260924 04:36:20.217993 7859 raft_consensus.cc:2258] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:20.218850 7859 ts_tablet_manager.cc:1905] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:20.219314 7858 raft_consensus.cc:2229] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:20.219470 7858 raft_consensus.cc:2258] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:20.220202 7858 ts_tablet_manager.cc:1905] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:20.221076 7859 ts_tablet_manager.cc:1918] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:20.221189 7859 log.cc:1196] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/765cc639701a4943ad0e86458b0f5c59
I20260924 04:36:20.221455 7859 ts_tablet_manager.cc:1939] T 765cc639701a4943ad0e86458b0f5c59 P b574eca291e34af69b610f262ea5dc21: Deleting consensus metadata
I20260924 04:36:20.222045 7275 catalog_manager.cc:4841] TS b574eca291e34af69b610f262ea5dc21 (127.0.98.2:33661): tablet 765cc639701a4943ad0e86458b0f5c59 (table test-table [id=c24d1daf4a8940cd970cd65d84ba9a34]) successfully deleted
I20260924 04:36:20.223434 7858 ts_tablet_manager.cc:1918] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:20.223515 7858 log.cc:1196] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/765cc639701a4943ad0e86458b0f5c59
I20260924 04:36:20.223781 7858 ts_tablet_manager.cc:1939] T 765cc639701a4943ad0e86458b0f5c59 P 9c83f256d94d401bb7d3b788ab2e248e: Deleting consensus metadata
I20260924 04:36:20.224429 7275 catalog_manager.cc:4841] TS 9c83f256d94d401bb7d3b788ab2e248e (127.0.98.3:40723): tablet 765cc639701a4943ad0e86458b0f5c59 (table test-table [id=c24d1daf4a8940cd970cd65d84ba9a34]) successfully deleted
I20260924 04:36:20.226295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 7476
I20260924 04:36:20.226619 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:20.226314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40885 (local address 127.0.0.1:43972)
0924 04:36:20.226456 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:20.226519 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 04:36:20.227674 7276 catalog_manager.cc:4584] TS e4029b6d160547fdb897b6ee50f4b9ec (127.0.98.1:40885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 765cc639701a4943ad0e86458b0f5c59: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40885: connect: Connection refused (error 111)
I20260924 04:36:20.233253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 7612
I20260924 04:36:20.238721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 6968
I20260924 04:36:20.243415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 7071
2026-09-24T04:36:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5528 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:20 dist-test-slave-jlnt krb5kdc[7862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:20 dist-test-slave-jlnt krb5kdc[7862](info): set up 2 sockets
Sep 24 04:36:20 dist-test-slave-jlnt krb5kdc[7862](info): commencing operation
krb5kdc: starting...
W20260924 04:36:21.285761 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:21 dist-test-slave-jlnt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:36:21.402340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45109
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:21.507634 7875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:21.507885 7875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:21.507922 7875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:21.511924 7875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:21.511977 7875 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:36:21.511994 7875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:21.512017 7875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:21.512068 7875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:21.512100 7875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:21.517105 7875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45109
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:21.518152 7875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:21.519160 7875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:21.527354 7881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:21.527370 7883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:21.527503 7875 server_base.cc:1034] running on GCE node
W20260924 04:36:21.527370 7880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:21.528239 7875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:21.529289 7875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:21.530454 7875 hybrid_clock.cc:648] HybridClock initialized: now 1790224581530436 us; error 32 us; skew 500 ppm
Sep 24 04:36:21 dist-test-slave-jlnt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:21.542920 7875 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:36:21.543816 7875 webserver.cc:460] Webserver started at http://127.0.98.62:39637/ using document root <none> and password file <none>
I20260924 04:36:21.544342 7875 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:21.544399 7875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:21.544555 7875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:21.546545 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "bc55a982086f481c88a40924e13ea2fa"
format_stamp: "Formatted at 2026-09-24 04:36:21 on dist-test-slave-jlnt"
server_key: "b1a53ad0d7f22a077d95f47c3d477424"
server_key_iv: "3899bd6395cc8e296e557c3522053e8b"
server_key_version: "encryptionkey@0"
I20260924 04:36:21.547075 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc55a982086f481c88a40924e13ea2fa"
format_stamp: "Formatted at 2026-09-24 04:36:21 on dist-test-slave-jlnt"
server_key: "b1a53ad0d7f22a077d95f47c3d477424"
server_key_iv: "3899bd6395cc8e296e557c3522053e8b"
server_key_version: "encryptionkey@0"
I20260924 04:36:21.550565 7875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:21.552716 7890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:21.553894 7875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:21.553989 7875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "bc55a982086f481c88a40924e13ea2fa"
format_stamp: "Formatted at 2026-09-24 04:36:21 on dist-test-slave-jlnt"
server_key: "b1a53ad0d7f22a077d95f47c3d477424"
server_key_iv: "3899bd6395cc8e296e557c3522053e8b"
server_key_version: "encryptionkey@0"
I20260924 04:36:21.554064 7875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:21.585564 7875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:21.588469 7875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:21.588618 7875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:21.596015 7875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45109
I20260924 04:36:21.596053 7952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45109 every 8 connection(s)
I20260924 04:36:21.596915 7875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:21.597647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 7875
I20260924 04:36:21.597795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:21.598068 392 external_mini_cluster.cc:1428] Setting key 9b8f10fafdd8002d57bfde56176d5e0e
I20260924 04:36:21.599961 7953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:21.606398 7953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa: Bootstrap starting.
Sep 24 04:36:21 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:21.608556 7953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:21.609679 7953 log.cc:824] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:21.611542 7953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa: No bootstrap required, opened a new log
I20260924 04:36:21.611776 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:21.599331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.0.98.62:45109)
0924 04:36:21.599724 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:21.599743 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:21.599782 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:36:21.600440 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:21.600457 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:21.600779 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:21.601169 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:21.601981 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.602956 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.604025 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.604459 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.604574 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:21.607363 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:21.607368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:21.607380 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:21.607404 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:21.609110 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:21.609577 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:21.609583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:21.609590 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:21.609706 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:21.609986 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:21.609989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:21.609990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:21.610347 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:21.610534 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:21.610798 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 04:36:21.611058 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":137,"threads_started":1}
I20260924 04:36:21.614889 7953 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc55a982086f481c88a40924e13ea2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } }
I20260924 04:36:21.615121 7953 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:21.615177 7953 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc55a982086f481c88a40924e13ea2fa, State: Initialized, Role: FOLLOWER
I20260924 04:36:21.615622 7953 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc55a982086f481c88a40924e13ea2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } }
I20260924 04:36:21.615808 7953 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:21.615880 7953 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:21.615972 7953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:21.616736 7953 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc55a982086f481c88a40924e13ea2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } }
I20260924 04:36:21.617005 7953 leader_election.cc:302] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc55a982086f481c88a40924e13ea2fa; no voters:
I20260924 04:36:21.617295 7953 leader_election.cc:288] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:21.617408 7958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:21.617619 7958 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [term 1 LEADER]: Becoming Leader. State: Replica: bc55a982086f481c88a40924e13ea2fa, State: Running, Role: LEADER
I20260924 04:36:21.617901 7958 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc55a982086f481c88a40924e13ea2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } }
I20260924 04:36:21.618439 7953 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:21.619354 7959 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc55a982086f481c88a40924e13ea2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc55a982086f481c88a40924e13ea2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } } }
I20260924 04:36:21.619696 7959 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:21.619854 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc55a982086f481c88a40924e13ea2fa. Latest consensus state: current_term: 1 leader_uuid: "bc55a982086f481c88a40924e13ea2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc55a982086f481c88a40924e13ea2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } } }
I20260924 04:36:21.619957 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:21.620324 7967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:21.623111 7967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:21.628453 7967 catalog_manager.cc:1347] Generated new cluster ID: 05db7e8f330949ce84187aad5827dfd6
I20260924 04:36:21.628518 7967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:21.635025 7967 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:21.636260 7967 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:21.641851 7967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa: Generated new TSK 0
I20260924 04:36:21.642670 7967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:36:21.675801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45109
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:21.784178 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:21.784404 7979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:21.784444 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:21.788234 7979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:21.788282 7979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:21.788319 7979 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:36:21.788350 7979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:21.793275 7979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:21.794127 7979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:21.795013 7979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:36:21.804168 7979 server_base.cc:1034] running on GCE node
W20260924 04:36:21.804121 7984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:21.804137 7985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:21.804121 7987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:21.804677 7979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:21.805295 7979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:21.806442 7979 hybrid_clock.cc:648] HybridClock initialized: now 1790224581806429 us; error 23 us; skew 500 ppm
Sep 24 04:36:21 dist-test-slave-jlnt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:21.808928 7979 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:36:21.809746 7979 webserver.cc:460] Webserver started at http://127.0.98.1:34273/ using document root <none> and password file <none>
I20260924 04:36:21.810214 7979 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:21.810268 7979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:21.810418 7979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:21.812307 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "35854587ddfb4f88ac5bca4bbc099103"
format_stamp: "Formatted at 2026-09-24 04:36:21 on dist-test-slave-jlnt"
server_key: "a39781e8e49f064ea1f310ba82229f7e"
server_key_iv: "a2c426d1decbbf0dc34f12234bfbc4ca"
server_key_version: "encryptionkey@0"
I20260924 04:36:21.812795 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35854587ddfb4f88ac5bca4bbc099103"
format_stamp: "Formatted at 2026-09-24 04:36:21 on dist-test-slave-jlnt"
server_key: "a39781e8e49f064ea1f310ba82229f7e"
server_key_iv: "a2c426d1decbbf0dc34f12234bfbc4ca"
server_key_version: "encryptionkey@0"
I20260924 04:36:21.816252 7979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:36:21.818538 7994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:21.819849 7979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:21.819958 7979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "35854587ddfb4f88ac5bca4bbc099103"
format_stamp: "Formatted at 2026-09-24 04:36:21 on dist-test-slave-jlnt"
server_key: "a39781e8e49f064ea1f310ba82229f7e"
server_key_iv: "a2c426d1decbbf0dc34f12234bfbc4ca"
server_key_version: "encryptionkey@0"
I20260924 04:36:21.820091 7979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:21.837039 7979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:21.839686 7979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:21.839826 7979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:21.841226 7979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:21.841290 7979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:21.841352 7979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:21.841399 7979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:36:21 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:21.852056 7979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38093
I20260924 04:36:21.852146 8109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38093 every 8 connection(s)
I20260924 04:36:21.852926 7979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:21.854830 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:21.842023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43041 (local address 127.0.98.62:45109)
0924 04:36:21.842147 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:21.842152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:21.843106 (+ 954us) server_negotiation.cc:407] Connection header received
0924 04:36:21.844045 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:21.844051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:21.844131 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:21.844229 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:21.845523 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.846285 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.847708 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.848001 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.848121 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:21.851374 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:21.851378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:21.851381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:21.851406 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:21.852981 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:21.853515 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:21.853518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:21.853520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:21.853604 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:21.853930 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:21.853933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:21.853934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:21.854109 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:21.854187 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:21.854620 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 04:36:21.854716 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:36:21.855520 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:21.842529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.98.1:43041)
0924 04:36:21.842971 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:21.843009 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:21.843807 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:21.844446 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:21.844454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:21.844779 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:21.845374 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:21.845390 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.846400 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.846404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:21.847577 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:21.847587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.848164 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.848168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:21.848367 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:21.848894 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:21.848908 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:21.851206 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:21.853082 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:21.853087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:21.853101 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:21.853399 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:21.853716 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:21.853719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:21.853720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:21.853842 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:21.854197 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:21.854201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:21.854400 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 04:36:21.854534 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 04:36:21.854769 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":132,"threads_started":1}
I20260924 04:36:21.856755 8110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45109
I20260924 04:36:21.856902 8110 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:21.857404 8110 heartbeater.cc:511] Master 127.0.98.62:45109 requested a full tablet report, sending...
I20260924 04:36:21.858958 7906 ts_manager.cc:194] Registered new tserver with Master: 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093)
I20260924 04:36:21.860055 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43041
I20260924 04:36:21.861908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 7979
I20260924 04:36:21.862039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:21.862389 392 external_mini_cluster.cc:1428] Setting key 89bdabc2ceb52c648bd93a90a808b554
I20260924 04:36:21.865989 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:21.858899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58349 (local address 127.0.98.62:45109)
0924 04:36:21.859006 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:21.859009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:21.859237 (+ 228us) server_negotiation.cc:407] Connection header received
0924 04:36:21.859358 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:21.859361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:21.859403 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:21.859488 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:21.859881 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.860452 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.861293 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.861508 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.861612 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:21.862739 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:21.862742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:21.862744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:21.862763 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:21.864374 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:21.864762 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:21.864768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:21.864772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:21.864857 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:21.865095 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:21.865101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:21.865105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:21.865353 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:21.865461 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:21.865752 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 04:36:21.865881 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:36:21.866086 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:21.858818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.98.1:58349)
0924 04:36:21.859161 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:21.859173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:21.859274 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:21.859502 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:21.859504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:21.859690 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:21.859775 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:21.859780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.860570 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.860573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:21.861126 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:21.861182 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:21.861616 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:21.861619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:21.861718 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:21.862206 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:21.862222 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:21.862620 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:21.864484 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:21.864486 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:21.864488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:21.864670 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:21.864945 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:21.864947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:21.864948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:21.865014 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:21.865464 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:21.865469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:21.865584 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 04:36:21.865704 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 04:36:21.865847 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:36:21.888451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45109
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:21.997112 8119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:21.997443 8119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:21.997485 8119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:22.001384 8119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:22.001457 8119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:22.001525 8119 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:36:22.001600 8119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:22.006860 8119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:22.007794 8119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:22.008724 8119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:22.018170 8125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:22.018146 8124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:22.018152 8127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:22.018321 8119 server_base.cc:1034] running on GCE node
I20260924 04:36:22.018821 8119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:22.019539 8119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:22.020699 8119 hybrid_clock.cc:648] HybridClock initialized: now 1790224582020670 us; error 42 us; skew 500 ppm
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:22.023667 8119 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:36:22.024560 8119 webserver.cc:460] Webserver started at http://127.0.98.2:44689/ using document root <none> and password file <none>
I20260924 04:36:22.025071 8119 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:22.025126 8119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:22.025359 8119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:22.027313 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "78b6bf862ff74d2799096f4985afb3e1"
format_stamp: "Formatted at 2026-09-24 04:36:22 on dist-test-slave-jlnt"
server_key: "df36f4de8a252b8a9e6df67fa4f2c783"
server_key_iv: "ee40c6aaf56f9f6646465e90ae607a01"
server_key_version: "encryptionkey@0"
I20260924 04:36:22.027843 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78b6bf862ff74d2799096f4985afb3e1"
format_stamp: "Formatted at 2026-09-24 04:36:22 on dist-test-slave-jlnt"
server_key: "df36f4de8a252b8a9e6df67fa4f2c783"
server_key_iv: "ee40c6aaf56f9f6646465e90ae607a01"
server_key_version: "encryptionkey@0"
I20260924 04:36:22.031437 8119 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:36:22.033870 8134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:22.035068 8119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:22.035203 8119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "78b6bf862ff74d2799096f4985afb3e1"
format_stamp: "Formatted at 2026-09-24 04:36:22 on dist-test-slave-jlnt"
server_key: "df36f4de8a252b8a9e6df67fa4f2c783"
server_key_iv: "ee40c6aaf56f9f6646465e90ae607a01"
server_key_version: "encryptionkey@0"
I20260924 04:36:22.035312 8119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:22.055753 8119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:22.058776 8119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:22.058974 8119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:22.060433 8119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:22.060506 8119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:22.060571 8119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:22.060623 8119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:22.071346 8119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38203
I20260924 04:36:22.071372 8249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38203 every 8 connection(s)
I20260924 04:36:22.072441 8119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:22.073921 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.061010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50943 (local address 127.0.98.62:45109)
0924 04:36:22.061130 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.061159 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.062057 (+ 898us) server_negotiation.cc:407] Connection header received
0924 04:36:22.063076 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.063079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.063148 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.063196 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:22.064588 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.065513 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.066752 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.066991 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.067117 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.070106 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:22.070110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:22.070112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:22.070138 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:22.071791 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.072288 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.072292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.072294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.072387 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.072776 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.072781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.072783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.073036 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:22.073165 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.073690 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.073792 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:36:22.074426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 8119
I20260924 04:36:22.074597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:22.074779 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.061481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.98.2:50943)
0924 04:36:22.061909 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.061967 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.062831 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.063499 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.063506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.063902 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:22.064415 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.064429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.065648 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.065652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.066625 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.066634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.067185 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.067194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.067436 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.068102 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:22.068128 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:22.069946 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:22.071920 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.071924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.071935 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.072183 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.072511 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.072517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.072520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.072675 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.073222 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:22.073230 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:22.073486 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.073641 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.073879 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
I20260924 04:36:22.074940 392 external_mini_cluster.cc:1428] Setting key f51cdef4a00f01a0b447dc558ed8eda9
I20260924 04:36:22.076198 8250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45109
I20260924 04:36:22.076355 8250 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:22.076906 8250 heartbeater.cc:511] Master 127.0.98.62:45109 requested a full tablet report, sending...
I20260924 04:36:22.077992 7906 ts_manager.cc:194] Registered new tserver with Master: 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203)
I20260924 04:36:22.078806 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50943
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:36:22.085601 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.078505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38231 (local address 127.0.98.62:45109)
0924 04:36:22.078620 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.078624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.078908 (+ 284us) server_negotiation.cc:407] Connection header received
0924 04:36:22.079076 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.079081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.079136 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.079251 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:22.079710 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.080241 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.081047 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.081364 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.081471 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.082566 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:22.082569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:22.082572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:22.082593 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:22.083959 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.084367 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.084371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.084373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.084450 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.084730 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.084734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.084736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.084935 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:22.085043 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.085341 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.085447 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:36:22.085610 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.078413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.98.2:38231)
0924 04:36:22.078829 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.078845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.078970 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.079265 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.079268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.079471 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:22.079600 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.079607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.080364 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.080367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.080930 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.080938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.081474 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.081477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.081573 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.082121 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:22.082131 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:22.082437 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:22.084068 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.084071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.084073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.084268 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.084546 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.084549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.084550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.084634 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.085033 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:22.085036 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:22.085129 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.085279 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.085421 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":207,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:36:22.104138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45109
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:22.211786 8259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:22.212049 8259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:22.212117 8259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:22.216362 8259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:22.216442 8259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:22.216506 8259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:36:22.216589 8259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:22.222151 8259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:22.223204 8259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:22.224218 8259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:22.233641 8265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:22.233662 8264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:22.233757 8259 server_base.cc:1034] running on GCE node
W20260924 04:36:22.233708 8267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:22.234220 8259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:22.234863 8259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:22.236042 8259 hybrid_clock.cc:648] HybridClock initialized: now 1790224582236026 us; error 27 us; skew 500 ppm
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:22.238893 8259 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:36:22.239851 8259 webserver.cc:460] Webserver started at http://127.0.98.3:46175/ using document root <none> and password file <none>
I20260924 04:36:22.240392 8259 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:22.240478 8259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:22.240691 8259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:22.242712 8259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "8f03a802c8f34250b81fa5392eee1915"
format_stamp: "Formatted at 2026-09-24 04:36:22 on dist-test-slave-jlnt"
server_key: "636a18dcecc3829852421ada0e7f1ca8"
server_key_iv: "1e9f1c955a975a9eedd44adb32dd5b72"
server_key_version: "encryptionkey@0"
I20260924 04:36:22.243252 8259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8f03a802c8f34250b81fa5392eee1915"
format_stamp: "Formatted at 2026-09-24 04:36:22 on dist-test-slave-jlnt"
server_key: "636a18dcecc3829852421ada0e7f1ca8"
server_key_iv: "1e9f1c955a975a9eedd44adb32dd5b72"
server_key_version: "encryptionkey@0"
I20260924 04:36:22.246817 8259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:22.249096 8274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:22.250334 8259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:22.250470 8259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "8f03a802c8f34250b81fa5392eee1915"
format_stamp: "Formatted at 2026-09-24 04:36:22 on dist-test-slave-jlnt"
server_key: "636a18dcecc3829852421ada0e7f1ca8"
server_key_iv: "1e9f1c955a975a9eedd44adb32dd5b72"
server_key_version: "encryptionkey@0"
I20260924 04:36:22.250573 8259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:22.266140 8259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:22.269366 8259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:22.269562 8259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:22.270967 8259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:22.271036 8259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:22.271108 8259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:22.271149 8259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:22.281703 8259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45369
I20260924 04:36:22.282507 8259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:22.282454 8389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45369 every 8 connection(s)
I20260924 04:36:22.284309 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.271748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60341 (local address 127.0.98.62:45109)
0924 04:36:22.271867 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.271870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.272849 (+ 979us) server_negotiation.cc:407] Connection header received
0924 04:36:22.274025 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.274032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.274104 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.274213 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:22.275345 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.276047 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.277543 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.277774 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.277893 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.280727 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:22.280731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:22.280733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:22.280757 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:22.282282 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.282873 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.282878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.282880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.282994 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.283381 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.283383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.283384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.283556 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:22.283635 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.284093 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.284198 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:36:22.285097 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.272253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.98.3:60341)
0924 04:36:22.272710 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.272749 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.273733 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.274422 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.274431 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.274746 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:22.275186 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.275222 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.276171 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.276175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.277400 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.277409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.277935 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.277940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.278111 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.278654 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:22.278669 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:22.280558 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:22.282417 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.282424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.282438 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.282766 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.283106 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.283109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.283111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.283269 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.283671 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:22.283676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:22.283881 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.284051 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.284274 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":131,"threads_started":1}
I20260924 04:36:22.286257 8390 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45109
I20260924 04:36:22.286412 8390 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:22.286870 8390 heartbeater.cc:511] Master 127.0.98.62:45109 requested a full tablet report, sending...
I20260924 04:36:22.287834 7907 ts_manager.cc:194] Registered new tserver with Master: 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3:45369)
I20260924 04:36:22.288439 7907 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60341
I20260924 04:36:22.290197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 8259
I20260924 04:36:22.290316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:22.290625 392 external_mini_cluster.cc:1428] Setting key 494032f6c6e9a8b2786830f024553682
I20260924 04:36:22.292501 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:36:22.295632 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.288390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.98.3:43729)
0924 04:36:22.288799 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.288820 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.288892 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.289109 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.289112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.289327 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:22.289431 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.289438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.290154 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.290157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.290843 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.290851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.291407 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.291411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.291499 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.291987 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:22.292000 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:22.292304 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:22.294153 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.294157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.294159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.294424 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.294699 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.294703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.294706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.294805 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.295131 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:22.295134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:22.295237 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.295351 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.295490 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":105,"threads_started":1}
I20260924 04:36:22.295778 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.288467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43729 (local address 127.0.98.62:45109)
0924 04:36:22.288606 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.288610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.288886 (+ 276us) server_negotiation.cc:407] Connection header received
0924 04:36:22.288954 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.288958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.289000 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.289073 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:22.289566 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.290035 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.290969 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.291282 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.291388 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.292443 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:22.292447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:22.292450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:22.292476 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:22.293976 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.294534 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.294536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.294537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.294610 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.294893 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.294895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.294896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.295040 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:22.295111 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.295423 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.295542 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224582, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:36:22.315562 7956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:22.309876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.0.98.62:45109)
0924 04:36:22.309995 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.309998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.310010 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:36:22.310045 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.310047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.310084 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.310151 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:22.310809 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.311305 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.312089 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.312301 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.312377 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.315419 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59440: BlockingRecv error: recv got EOF from 127.0.0.1:59440 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224582, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:22.324869 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.316774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.0.98.62:45109)
0924 04:36:22.316875 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.316878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.317023 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:36:22.317170 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.317173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.317218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.317302 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:22.317868 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.318363 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.319250 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.319454 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.319582 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.321933 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:22.321936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:22.321938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:22.321958 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:22.323198 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.323664 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.323669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.323671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.323757 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.324027 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.324030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.324031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.324198 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:22.324334 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.324602 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.324717 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:36:22.327489 7907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:36:22.330597 7907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:22.344880 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.338604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38093 (local address 127.0.0.1:45774)
0924 04:36:22.339766 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.339779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.339928 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.340597 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.340599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.340616 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:22.340763 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.340769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.342074 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.342077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.343424 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.343470 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.344280 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.344283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.344435 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.344493 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.344604 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.344729 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":48,"thread_start_us":82,"threads_started":1}
I20260924 04:36:22.344890 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.339467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.0.98.2:38203)
0924 04:36:22.340150 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.340167 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.340187 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:36:22.340253 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.340257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.340450 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.340590 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:22.340865 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.341715 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.343505 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.344135 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.344264 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.344435 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.344652 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.344747 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":109,"threads_started":1}
I20260924 04:36:22.344883 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.339304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38203 (local address 127.0.0.1:44868)
0924 04:36:22.339962 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.339976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.340042 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.340588 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.340591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.340606 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:22.340760 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.340766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.341857 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.341860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.343371 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.343379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.344264 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.344268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.344399 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.344505 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.344620 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.344744 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":344,"thread_start_us":91,"threads_started":1}
I20260924 04:36:22.345338 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.339293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.0.98.1:38093)
0924 04:36:22.339604 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.339619 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.339857 (+ 238us) server_negotiation.cc:407] Connection header received
0924 04:36:22.340084 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.340090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.340310 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.340427 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:22.340893 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.341969 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.343627 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.344154 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.344280 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.344441 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.345072 (+ 631us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.345202 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260924 04:36:22.345839 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.338426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45369 (local address 127.0.0.1:52532)
0924 04:36:22.338939 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.338989 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.339114 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.340201 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.340207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.340232 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:22.340351 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.340356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.341813 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.341816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.343501 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.343508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.345453 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.345458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.345558 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.345580 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.345662 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.345725 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":98,"thread_start_us":68,"threads_started":1}
I20260924 04:36:22.345844 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.338596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.0.98.3:45369)
0924 04:36:22.339352 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.339370 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.339394 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:36:22.339471 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.339476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.339692 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.339850 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:22.340473 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.341687 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.344500 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.345325 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.345439 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.345588 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.345682 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.345745 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":120,"threads_started":1}
I20260924 04:36:22.348357 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 318179b641e74829a864e679421f7d3b (DEFAULT_TABLE table=test-table [id=6e8a56ffa2c646929e77355a8ee5474e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.348367 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 318179b641e74829a864e679421f7d3b (DEFAULT_TABLE table=test-table [id=6e8a56ffa2c646929e77355a8ee5474e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.348898 8167 tablet_service.cc:1467] Processing CreateTablet for tablet 318179b641e74829a864e679421f7d3b (DEFAULT_TABLE table=test-table [id=6e8a56ffa2c646929e77355a8ee5474e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.349392 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318179b641e74829a864e679421f7d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.349404 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318179b641e74829a864e679421f7d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.349785 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318179b641e74829a864e679421f7d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.356336 8418 tablet_bootstrap.cc:492] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: Bootstrap starting.
I20260924 04:36:22.357942 8419 tablet_bootstrap.cc:492] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: Bootstrap starting.
I20260924 04:36:22.358642 8418 tablet_bootstrap.cc:654] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.359367 8418 log.cc:824] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:22.359709 8419 tablet_bootstrap.cc:654] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.360044 8420 tablet_bootstrap.cc:492] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: Bootstrap starting.
I20260924 04:36:22.360457 8419 log.cc:824] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:22.361403 8418 tablet_bootstrap.cc:492] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: No bootstrap required, opened a new log
I20260924 04:36:22.361559 8418 ts_tablet_manager.cc:1397] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:36:22.362226 8420 tablet_bootstrap.cc:654] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.362841 8420 log.cc:824] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:22.364703 8420 tablet_bootstrap.cc:492] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: No bootstrap required, opened a new log
I20260924 04:36:22.364708 8419 tablet_bootstrap.cc:492] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: No bootstrap required, opened a new log
I20260924 04:36:22.364579 8418 raft_consensus.cc:348] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } }
I20260924 04:36:22.364802 8418 raft_consensus.cc:374] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.364861 8418 raft_consensus.cc:729] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f03a802c8f34250b81fa5392eee1915, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.364890 8420 ts_tablet_manager.cc:1397] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 04:36:22.364890 8419 ts_tablet_manager.cc:1397] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 04:36:22.365309 8418 consensus_queue.cc:260] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } }
I20260924 04:36:22.365921 8390 heartbeater.cc:503] Master 127.0.98.62:45109 was elected leader, sending a full tablet report...
I20260924 04:36:22.366286 8418 ts_tablet_manager.cc:1428] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:36:22.368788 8420 raft_consensus.cc:348] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } }
I20260924 04:36:22.369040 8420 raft_consensus.cc:374] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.369102 8420 raft_consensus.cc:729] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35854587ddfb4f88ac5bca4bbc099103, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.369050 8419 raft_consensus.cc:348] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } }
I20260924 04:36:22.369297 8419 raft_consensus.cc:374] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.369356 8419 raft_consensus.cc:729] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b6bf862ff74d2799096f4985afb3e1, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.369669 8420 consensus_queue.cc:260] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } }
I20260924 04:36:22.369711 8419 consensus_queue.cc:260] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } }
I20260924 04:36:22.370285 8110 heartbeater.cc:503] Master 127.0.98.62:45109 was elected leader, sending a full tablet report...
I20260924 04:36:22.370603 8419 ts_tablet_manager.cc:1428] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:36:22.370603 8420 ts_tablet_manager.cc:1428] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:36:22.370934 8250 heartbeater.cc:503] Master 127.0.98.62:45109 was elected leader, sending a full tablet report...
I20260924 04:36:22.377712 8425 raft_consensus.cc:482] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:22.377838 8425 raft_consensus.cc:504] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } }
I20260924 04:36:22.378722 8425 leader_election.cc:288] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3:45369), 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203)
I20260924 04:36:22.382688 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.379004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40077 (local address 127.0.98.3:45369)
0924 04:36:22.379119 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.379123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.379335 (+ 212us) server_negotiation.cc:407] Connection header received
0924 04:36:22.379486 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.379489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.379542 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.379619 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:22.379976 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.380570 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.381699 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.382133 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.382212 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.382289 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.382490 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.382564 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:36:22.382705 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.379004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47607 (local address 127.0.98.2:38203)
0924 04:36:22.379209 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.379212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.379713 (+ 501us) server_negotiation.cc:407] Connection header received
0924 04:36:22.379822 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.379826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.379871 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.379954 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:22.380204 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.380699 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.381786 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.382192 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.382296 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.382354 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.382544 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.382609 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260924 04:36:22.382715 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.378925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45369 (local address 127.0.98.1:40077)
0924 04:36:22.379245 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.379256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.379384 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.379643 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.379645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.379698 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:22.379866 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.379874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.380677 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.380681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.381581 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.381590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.382259 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.382266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.382357 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.382387 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.382452 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.382535 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":90,"threads_started":1}
I20260924 04:36:22.382723 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.378925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38203 (local address 127.0.98.1:47607)
0924 04:36:22.379629 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.379642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.379724 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.379975 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.379978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.379993 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:22.380099 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.380105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.380804 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.380807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.381669 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.381681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.382329 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.382332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.382406 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.382422 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.382516 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.382586 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":12,"thread_start_us":77,"threads_started":1}
I20260924 04:36:22.383315 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318179b641e74829a864e679421f7d3b" candidate_uuid: "35854587ddfb4f88ac5bca4bbc099103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f03a802c8f34250b81fa5392eee1915" is_pre_election: true
I20260924 04:36:22.383626 8344 raft_consensus.cc:2454] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35854587ddfb4f88ac5bca4bbc099103 in term 0.
I20260924 04:36:22.383870 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318179b641e74829a864e679421f7d3b" candidate_uuid: "35854587ddfb4f88ac5bca4bbc099103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6bf862ff74d2799096f4985afb3e1" is_pre_election: true
I20260924 04:36:22.384107 7997 leader_election.cc:302] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35854587ddfb4f88ac5bca4bbc099103, 8f03a802c8f34250b81fa5392eee1915; no voters:
I20260924 04:36:22.384124 8204 raft_consensus.cc:2454] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35854587ddfb4f88ac5bca4bbc099103 in term 0.
I20260924 04:36:22.384361 8425 raft_consensus.cc:2781] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:22.384477 8425 raft_consensus.cc:482] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:22.384539 8425 raft_consensus.cc:3037] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.385725 8425 raft_consensus.cc:504] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } }
I20260924 04:36:22.386304 8425 leader_election.cc:288] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [CANDIDATE]: Term 1 election: Requested vote from peers 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3:45369), 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203)
I20260924 04:36:22.386366 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318179b641e74829a864e679421f7d3b" candidate_uuid: "35854587ddfb4f88ac5bca4bbc099103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f03a802c8f34250b81fa5392eee1915"
I20260924 04:36:22.386442 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318179b641e74829a864e679421f7d3b" candidate_uuid: "35854587ddfb4f88ac5bca4bbc099103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6bf862ff74d2799096f4985afb3e1"
I20260924 04:36:22.386479 8344 raft_consensus.cc:3037] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.386544 8204 raft_consensus.cc:3037] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.387532 8344 raft_consensus.cc:2454] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35854587ddfb4f88ac5bca4bbc099103 in term 1.
I20260924 04:36:22.387817 7997 leader_election.cc:302] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35854587ddfb4f88ac5bca4bbc099103, 8f03a802c8f34250b81fa5392eee1915; no voters:
I20260924 04:36:22.387930 8204 raft_consensus.cc:2454] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35854587ddfb4f88ac5bca4bbc099103 in term 1.
I20260924 04:36:22.388006 8425 raft_consensus.cc:2781] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:22.388271 8425 raft_consensus.cc:686] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 1 LEADER]: Becoming Leader. State: Replica: 35854587ddfb4f88ac5bca4bbc099103, State: Running, Role: LEADER
I20260924 04:36:22.388619 8425 consensus_queue.cc:237] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } }
I20260924 04:36:22.391650 7906 catalog_manager.cc:5495] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "35854587ddfb4f88ac5bca4bbc099103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } health_report { overall_health: UNKNOWN } } }
I20260924 04:36:22.406443 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.402893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.0.98.1:38093)
0924 04:36:22.403034 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.403039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.403053 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:36:22.403214 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.403217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.403278 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.403392 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:22.403632 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.404139 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.404984 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.405290 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.405400 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.405657 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:22.406069 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:22.406192 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.406256 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.406309 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:36:22.412253 8204 raft_consensus.cc:1261] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Refusing update from remote peer 35854587ddfb4f88ac5bca4bbc099103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:22.412622 8344 raft_consensus.cc:1261] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Refusing update from remote peer 35854587ddfb4f88ac5bca4bbc099103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:22.412953 8425 consensus_queue.cc:1035] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:22.413229 8429 consensus_queue.cc:1035] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:22.420433 8432 mvcc.cc:205] Tried to move back new op lower bound from 7332759889553649664 to 7332759889466302464. Current Snapshot: MvccSnapshot[applied={T|T < 7332759889553649664}]
I20260924 04:36:22.421072 8434 mvcc.cc:205] Tried to move back new op lower bound from 7332759889553649664 to 7332759889466302464. Current Snapshot: MvccSnapshot[applied={T|T < 7332759889553649664}]
I20260924 04:36:22.422560 8433 mvcc.cc:205] Tried to move back new op lower bound from 7332759889553649664 to 7332759889466302464. Current Snapshot: MvccSnapshot[applied={T|T < 7332759889553649664}]
I20260924 04:36:22.431170 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:22.431342 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:22.433557 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa: Sending DeleteTablet for 3 replicas of tablet 318179b641e74829a864e679421f7d3b
I20260924 04:36:22.434242 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet 318179b641e74829a864e679421f7d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52532
I20260924 04:36:22.434454 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet 318179b641e74829a864e679421f7d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260924 04:36:22.434571 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet 318179b641e74829a864e679421f7d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45774
I20260924 04:36:22.435045 8444 tablet_replica.cc:330] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: stopping tablet replica
I20260924 04:36:22.435144 8442 tablet_replica.cc:330] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: stopping tablet replica
I20260924 04:36:22.435338 8444 raft_consensus.cc:2229] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:22.435314 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:36:22.435406 8442 raft_consensus.cc:2229] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:22.435673 8444 raft_consensus.cc:2258] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:36:22.435690 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:22.435689 8442 raft_consensus.cc:2258] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:22.436245 8443 tablet_replica.cc:330] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: stopping tablet replica
I20260924 04:36:22.436486 8443 raft_consensus.cc:2229] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:22.436741 8443 raft_consensus.cc:2258] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:22.437885 8442 ts_tablet_manager.cc:1905] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:22.439854 8443 ts_tablet_manager.cc:1905] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:22.440299 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 31721a5425a147e787a767794c1094a2 (DEFAULT_TABLE table=test-table [id=9884ac54bd54478e820296917d29379b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.440544 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31721a5425a147e787a767794c1094a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.440569 8167 tablet_service.cc:1467] Processing CreateTablet for tablet 31721a5425a147e787a767794c1094a2 (DEFAULT_TABLE table=test-table [id=9884ac54bd54478e820296917d29379b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.440832 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31721a5425a147e787a767794c1094a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.441646 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 31721a5425a147e787a767794c1094a2 (DEFAULT_TABLE table=test-table [id=9884ac54bd54478e820296917d29379b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.441883 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31721a5425a147e787a767794c1094a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.443012 8420 tablet_bootstrap.cc:492] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: Bootstrap starting.
I20260924 04:36:22.443231 8419 tablet_bootstrap.cc:492] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: Bootstrap starting.
I20260924 04:36:22.443931 8420 tablet_bootstrap.cc:654] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.444056 8418 tablet_bootstrap.cc:492] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: Bootstrap starting.
I20260924 04:36:22.444118 8419 tablet_bootstrap.cc:654] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.444986 8418 tablet_bootstrap.cc:654] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.445048 8444 ts_tablet_manager.cc:1905] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:22.446189 8442 ts_tablet_manager.cc:1918] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:22.446228 8418 tablet_bootstrap.cc:492] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: No bootstrap required, opened a new log
I20260924 04:36:22.446288 8442 log.cc:1196] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/318179b641e74829a864e679421f7d3b
I20260924 04:36:22.446310 8418 ts_tablet_manager.cc:1397] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:22.446610 8442 ts_tablet_manager.cc:1939] T 318179b641e74829a864e679421f7d3b P 8f03a802c8f34250b81fa5392eee1915: Deleting consensus metadata
I20260924 04:36:22.446864 8418 raft_consensus.cc:348] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.447124 8418 raft_consensus.cc:374] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.447189 8418 raft_consensus.cc:729] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f03a802c8f34250b81fa5392eee1915, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.447317 8418 consensus_queue.cc:260] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.447609 8418 ts_tablet_manager.cc:1428] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:36:22.448400 7893 catalog_manager.cc:4841] TS 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3:45369): tablet 318179b641e74829a864e679421f7d3b (table test-table [id=6e8a56ffa2c646929e77355a8ee5474e]) successfully deleted
I20260924 04:36:22.448967 8443 ts_tablet_manager.cc:1918] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:22.449080 8443 log.cc:1196] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/318179b641e74829a864e679421f7d3b
I20260924 04:36:22.449433 8443 ts_tablet_manager.cc:1939] T 318179b641e74829a864e679421f7d3b P 78b6bf862ff74d2799096f4985afb3e1: Deleting consensus metadata
I20260924 04:36:22.450603 7894 catalog_manager.cc:4841] TS 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203): tablet 318179b641e74829a864e679421f7d3b (table test-table [id=6e8a56ffa2c646929e77355a8ee5474e]) successfully deleted
I20260924 04:36:22.451195 8444 ts_tablet_manager.cc:1918] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:22.451288 8444 log.cc:1196] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/318179b641e74829a864e679421f7d3b
I20260924 04:36:22.451458 8419 tablet_bootstrap.cc:492] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: No bootstrap required, opened a new log
I20260924 04:36:22.451541 8419 ts_tablet_manager.cc:1397] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 04:36:22.451592 8444 ts_tablet_manager.cc:1939] T 318179b641e74829a864e679421f7d3b P 35854587ddfb4f88ac5bca4bbc099103: Deleting consensus metadata
I20260924 04:36:22.451767 8420 tablet_bootstrap.cc:492] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: No bootstrap required, opened a new log
I20260924 04:36:22.451833 8420 ts_tablet_manager.cc:1397] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260924 04:36:22.451954 8419 raft_consensus.cc:348] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.452050 8419 raft_consensus.cc:374] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.452078 8419 raft_consensus.cc:729] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b6bf862ff74d2799096f4985afb3e1, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.452185 8419 consensus_queue.cc:260] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.452236 8420 raft_consensus.cc:348] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.452333 8420 raft_consensus.cc:374] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.452365 8420 raft_consensus.cc:729] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35854587ddfb4f88ac5bca4bbc099103, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.452414 8419 ts_tablet_manager.cc:1428] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:22.452468 8420 consensus_queue.cc:260] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.452661 7894 catalog_manager.cc:4841] TS 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093): tablet 318179b641e74829a864e679421f7d3b (table test-table [id=6e8a56ffa2c646929e77355a8ee5474e]) successfully deleted
I20260924 04:36:22.452725 8420 ts_tablet_manager.cc:1428] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:22.502997 8426 raft_consensus.cc:482] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:22.503151 8426 raft_consensus.cc:504] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.504359 8426 leader_election.cc:288] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093), 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3:45369)
I20260924 04:36:22.508855 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.504609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38093 (local address 127.0.98.2:59737)
0924 04:36:22.504792 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.504810 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.504896 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.505182 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.505185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.505204 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:22.505321 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.505327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.506426 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.506432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.507560 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.507570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.508432 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.508436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.508551 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.508576 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.508646 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.508714 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 04:36:22.508981 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.504706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59737 (local address 127.0.98.1:38093)
0924 04:36:22.504831 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.504835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.504850 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:22.505015 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.505020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.505074 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.505169 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:22.505751 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.506314 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.507698 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.508306 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.508416 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.508481 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.508790 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.508863 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:36:22.509477 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.504609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45369 (local address 127.0.98.2:36665)
0924 04:36:22.505183 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.505196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.505332 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.505636 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.505638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.505653 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:22.505782 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.505789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.506791 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.506794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.507918 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.507927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.508709 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.508713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.508809 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.508832 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.509261 (+ 429us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.509332 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":265,"thread_start_us":110,"threads_started":1}
I20260924 04:36:22.509509 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.504775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36665 (local address 127.0.98.3:45369)
0924 04:36:22.504885 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.504889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.505269 (+ 380us) server_negotiation.cc:407] Connection header received
0924 04:36:22.505455 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.505458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.505538 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.505625 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:22.505910 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.506671 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.508068 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.508560 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.508668 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.508722 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.509299 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.509387 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:36:22.509590 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31721a5425a147e787a767794c1094a2" candidate_uuid: "78b6bf862ff74d2799096f4985afb3e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35854587ddfb4f88ac5bca4bbc099103" is_pre_election: true
I20260924 04:36:22.509865 8064 raft_consensus.cc:2454] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b6bf862ff74d2799096f4985afb3e1 in term 0.
I20260924 04:36:22.510159 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31721a5425a147e787a767794c1094a2" candidate_uuid: "78b6bf862ff74d2799096f4985afb3e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f03a802c8f34250b81fa5392eee1915" is_pre_election: true
I20260924 04:36:22.510295 8344 raft_consensus.cc:2454] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b6bf862ff74d2799096f4985afb3e1 in term 0.
I20260924 04:36:22.510324 8138 leader_election.cc:302] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35854587ddfb4f88ac5bca4bbc099103, 78b6bf862ff74d2799096f4985afb3e1; no voters:
I20260924 04:36:22.510624 8426 raft_consensus.cc:2781] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:22.510697 8426 raft_consensus.cc:482] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:22.510736 8426 raft_consensus.cc:3037] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.511945 8426 raft_consensus.cc:504] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.512295 8426 leader_election.cc:288] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [CANDIDATE]: Term 1 election: Requested vote from peers 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093), 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3:45369)
I20260924 04:36:22.512630 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31721a5425a147e787a767794c1094a2" candidate_uuid: "78b6bf862ff74d2799096f4985afb3e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f03a802c8f34250b81fa5392eee1915"
I20260924 04:36:22.512668 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31721a5425a147e787a767794c1094a2" candidate_uuid: "78b6bf862ff74d2799096f4985afb3e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35854587ddfb4f88ac5bca4bbc099103"
I20260924 04:36:22.512751 8344 raft_consensus.cc:3037] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.512796 8064 raft_consensus.cc:3037] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.513784 8064 raft_consensus.cc:2454] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b6bf862ff74d2799096f4985afb3e1 in term 1.
I20260924 04:36:22.513803 8344 raft_consensus.cc:2454] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b6bf862ff74d2799096f4985afb3e1 in term 1.
I20260924 04:36:22.514047 8138 leader_election.cc:302] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35854587ddfb4f88ac5bca4bbc099103, 78b6bf862ff74d2799096f4985afb3e1; no voters:
I20260924 04:36:22.514241 8426 raft_consensus.cc:2781] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:22.514460 8426 raft_consensus.cc:686] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 1 LEADER]: Becoming Leader. State: Replica: 78b6bf862ff74d2799096f4985afb3e1, State: Running, Role: LEADER
I20260924 04:36:22.514711 8426 consensus_queue.cc:237] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.517048 7907 catalog_manager.cc:5495] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78b6bf862ff74d2799096f4985afb3e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } health_report { overall_health: UNKNOWN } } }
W20260924 04:36:22.533494 8391 tablet.cc:2369] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:36:22.573782 8251 tablet.cc:2369] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:22.580643 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.570936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.0.1:59452)
0924 04:36:22.571243 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.571255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.571392 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.571689 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.571692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.572116 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:22.572213 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.572220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.573315 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.573318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.574198 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.574211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.574877 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.574880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.574961 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.575699 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:22.575720 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:22.577458 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:22.579115 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.579125 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.579131 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.579351 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.579617 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:22.579621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:22.579623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:22.579704 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:22.580064 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:22.580070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:22.580183 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.580297 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.580446 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":117,"threads_started":1}
I20260924 04:36:22.580642 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.571020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.98.62:45109)
0924 04:36:22.571168 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.571173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.571335 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:36:22.571542 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.571544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.571583 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.571672 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:22.572337 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.573194 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.574325 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.574660 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.574736 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.577598 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:22.577601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:22.577606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:22.577630 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:22.579008 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.579449 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.579455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.579458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.579529 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.579795 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.579798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.579800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.579966 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:22.580068 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.580338 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.580501 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":53}
I20260924 04:36:22.584723 7907 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59452:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:36:22.585485 7907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:22.590312 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 36ddc34a75c4429ab9bfb847c3304b2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=763bdc34b2264a56b61d64b0d61b8199]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:36:22.590430 8167 tablet_service.cc:1467] Processing CreateTablet for tablet 36ddc34a75c4429ab9bfb847c3304b2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=763bdc34b2264a56b61d64b0d61b8199]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:36:22.590687 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ddc34a75c4429ab9bfb847c3304b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.590782 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ddc34a75c4429ab9bfb847c3304b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.592916 8420 tablet_bootstrap.cc:492] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: Bootstrap starting.
I20260924 04:36:22.593518 8419 tablet_bootstrap.cc:492] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: Bootstrap starting.
I20260924 04:36:22.593724 8420 tablet_bootstrap.cc:654] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.593843 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 36ddc34a75c4429ab9bfb847c3304b2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=763bdc34b2264a56b61d64b0d61b8199]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:36:22.594142 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ddc34a75c4429ab9bfb847c3304b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.594398 8419 tablet_bootstrap.cc:654] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.594838 8420 tablet_bootstrap.cc:492] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: No bootstrap required, opened a new log
I20260924 04:36:22.594918 8420 ts_tablet_manager.cc:1397] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:22.595283 8420 raft_consensus.cc:348] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.595387 8420 raft_consensus.cc:374] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.595418 8420 raft_consensus.cc:729] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35854587ddfb4f88ac5bca4bbc099103, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.595572 8419 tablet_bootstrap.cc:492] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: No bootstrap required, opened a new log
I20260924 04:36:22.595543 8420 consensus_queue.cc:260] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.595639 8419 ts_tablet_manager.cc:1397] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:22.595836 8420 ts_tablet_manager.cc:1428] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:22.595783 8425 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.595911 8425 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:36:22.596065 8419 raft_consensus.cc:348] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.596143 8419 raft_consensus.cc:374] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.596163 8419 raft_consensus.cc:729] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b6bf862ff74d2799096f4985afb3e1, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.596279 8419 consensus_queue.cc:260] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.596495 8419 ts_tablet_manager.cc:1428] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:22.596503 8426 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.596665 8426 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:36:22.601447 8418 tablet_bootstrap.cc:492] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: Bootstrap starting.
I20260924 04:36:22.602226 8418 tablet_bootstrap.cc:654] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.603471 8418 tablet_bootstrap.cc:492] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: No bootstrap required, opened a new log
I20260924 04:36:22.603605 8418 ts_tablet_manager.cc:1397] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:22.604004 8418 raft_consensus.cc:348] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.604148 8418 raft_consensus.cc:374] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.604195 8418 raft_consensus.cc:729] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f03a802c8f34250b81fa5392eee1915, State: Initialized, Role: FOLLOWER
W20260924 04:36:22.604214 8111 tablet.cc:2369] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:36:22.604326 8418 consensus_queue.cc:260] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.604624 8418 ts_tablet_manager.cc:1428] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:22.604868 8424 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.604978 8424 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:36:22.695879 8424 raft_consensus.cc:482] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:22.696015 8424 raft_consensus.cc:504] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.697120 8424 leader_election.cc:288] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093), 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203)
I20260924 04:36:22.701128 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.697518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48601 (local address 127.0.98.1:38093)
0924 04:36:22.697637 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.697641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.697655 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:36:22.697791 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.697794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.697849 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.697938 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:22.698641 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.699164 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.700200 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.700592 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.700717 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.700782 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.700937 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.701006 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:36:22.701128 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.697428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38093 (local address 127.0.98.3:48601)
0924 04:36:22.697552 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.697568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.697667 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.697972 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.697975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.697993 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:22.698506 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.698514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.699281 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.699287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.700077 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.700085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.700711 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.700715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.700814 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.700838 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.700926 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.700999 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":23}
I20260924 04:36:22.701295 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.697868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50617 (local address 127.0.98.2:38203)
0924 04:36:22.698013 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.698020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.698248 (+ 228us) server_negotiation.cc:407] Connection header received
0924 04:36:22.698369 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.698372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.698418 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.698502 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:22.698771 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.699336 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.700350 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.700732 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.700849 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.700907 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.701075 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.701161 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:36:22.701340 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.697744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38203 (local address 127.0.98.3:50617)
0924 04:36:22.698168 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.698179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.698284 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.698521 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.698523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.698539 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:22.698656 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.698661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.699444 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.699449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.700235 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.700243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.700851 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.700855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.700949 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.700970 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.701062 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.701130 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":3,"thread_start_us":160,"threads_started":1}
I20260924 04:36:22.701684 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ddc34a75c4429ab9bfb847c3304b2e" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35854587ddfb4f88ac5bca4bbc099103" is_pre_election: true
I20260924 04:36:22.701859 8064 raft_consensus.cc:2454] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 0.
I20260924 04:36:22.701910 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ddc34a75c4429ab9bfb847c3304b2e" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6bf862ff74d2799096f4985afb3e1" is_pre_election: true
I20260924 04:36:22.702054 8204 raft_consensus.cc:2454] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 0.
I20260924 04:36:22.702244 8278 leader_election.cc:302] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35854587ddfb4f88ac5bca4bbc099103, 8f03a802c8f34250b81fa5392eee1915; no voters:
I20260924 04:36:22.702484 8424 raft_consensus.cc:2781] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:22.702569 8424 raft_consensus.cc:482] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:22.702626 8424 raft_consensus.cc:3037] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.703598 8424 raft_consensus.cc:504] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.703934 8424 leader_election.cc:288] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 election: Requested vote from peers 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093), 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203)
I20260924 04:36:22.704357 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ddc34a75c4429ab9bfb847c3304b2e" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6bf862ff74d2799096f4985afb3e1"
I20260924 04:36:22.704347 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ddc34a75c4429ab9bfb847c3304b2e" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35854587ddfb4f88ac5bca4bbc099103"
I20260924 04:36:22.704463 8204 raft_consensus.cc:3037] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.704492 8064 raft_consensus.cc:3037] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.705507 8204 raft_consensus.cc:2454] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 1.
I20260924 04:36:22.705626 8064 raft_consensus.cc:2454] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 1.
I20260924 04:36:22.705828 8278 leader_election.cc:302] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b6bf862ff74d2799096f4985afb3e1, 8f03a802c8f34250b81fa5392eee1915; no voters:
I20260924 04:36:22.706085 8424 raft_consensus.cc:2781] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:22.706272 8424 raft_consensus.cc:686] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [term 1 LEADER]: Becoming Leader. State: Replica: 8f03a802c8f34250b81fa5392eee1915, State: Running, Role: LEADER
I20260924 04:36:22.706535 8424 consensus_queue.cc:237] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.708127 8461 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: TxnStatusTablet state changed. Reason: New leader 8f03a802c8f34250b81fa5392eee1915. Latest consensus state: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.708230 8461 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:36:22.708491 8463 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:36:22.708663 8463 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:36:22.709033 7906 catalog_manager.cc:5495] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } health_report { overall_health: HEALTHY } } }
I20260924 04:36:22.738842 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.735197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.0.98.3:45369)
0924 04:36:22.735335 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.735340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.735355 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:22.735552 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.735556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.735624 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.735737 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:22.735973 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.736486 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.737385 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.737612 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.737721 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.738013 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:22.738421 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:22.738538 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.738631 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.738681 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:36:22.738821 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.735100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45369 (local address 127.0.0.1:52540)
0924 04:36:22.735228 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.735246 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.735352 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.735746 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.735749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.735756 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:36:22.735865 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.735871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.736601 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.736605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.737269 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.737278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.737717 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.737721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.737821 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.737870 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:36:22.738523 (+ 653us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:36:22.738537 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.738618 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.738701 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 04:36:22.741830 8204 raft_consensus.cc:1261] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Refusing update from remote peer 8f03a802c8f34250b81fa5392eee1915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:22.741994 8064 raft_consensus.cc:1261] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Refusing update from remote peer 8f03a802c8f34250b81fa5392eee1915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:22.742414 8461 consensus_queue.cc:1035] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:22.742587 8424 consensus_queue.cc:1035] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:22.743947 8426 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: TxnStatusTablet state changed. Reason: New leader 8f03a802c8f34250b81fa5392eee1915. Latest consensus state: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.744053 8426 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:36:22.745220 8425 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: TxnStatusTablet state changed. Reason: New leader 8f03a802c8f34250b81fa5392eee1915. Latest consensus state: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.745332 8425 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:36:22.747974 8461 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.748090 8461 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:36:22.748270 8461 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.748364 8461 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 8f03a802c8f34250b81fa5392eee1915: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:36:22.748997 8425 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.749102 8425 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 35854587ddfb4f88ac5bca4bbc099103: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:36:22.750875 8426 tablet_replica.cc:439] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } } }
I20260924 04:36:22.750988 8426 tablet_replica.cc:442] T 36ddc34a75c4429ab9bfb847c3304b2e P 78b6bf862ff74d2799096f4985afb3e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:36:22.756590 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.752896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.0.98.2:38203)
0924 04:36:22.753001 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.753005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.753068 (+ 63us) server_negotiation.cc:407] Connection header received
0924 04:36:22.753249 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.753251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.753298 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.753412 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:22.753651 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.754232 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.755137 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.755367 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.755513 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.755776 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:22.756206 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:22.756327 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.756390 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.756439 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:36:22.767462 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.763468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46041 (local address 127.0.98.3:45369)
0924 04:36:22.763576 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.763580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.763839 (+ 259us) server_negotiation.cc:407] Connection header received
0924 04:36:22.763948 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.763950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.763990 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.764050 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:22.764712 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.765304 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.766521 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.766731 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.766804 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.767104 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:22.767164 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:22.767211 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.767325 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.767364 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:36:22.767541 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.763392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45369 (local address 127.0.98.2:46041)
0924 04:36:22.763758 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.763774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.763872 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.764080 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.764083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.764090 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:36:22.764600 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.764609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.765400 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.765403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.766403 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.766414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.766830 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.766834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.766926 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.766974 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:36:22.767237 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:36:22.767243 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.767310 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.767382 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":94,"threads_started":1}
I20260924 04:36:22.774220 8064 raft_consensus.cc:1261] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Refusing update from remote peer 78b6bf862ff74d2799096f4985afb3e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:22.774276 8344 raft_consensus.cc:1261] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Refusing update from remote peer 78b6bf862ff74d2799096f4985afb3e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:22.774668 8426 consensus_queue.cc:1035] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:22.774870 8449 consensus_queue.cc:1035] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:22.796563 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.792545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38203 (local address 127.0.98.3:54019)
0924 04:36:22.792718 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:22.792733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:22.792800 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:22.793258 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:22.793261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:22.793268 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:36:22.793384 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.793389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.794402 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.794406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.795359 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:22.795369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.795810 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.795813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:22.795911 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.795961 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:36:22.796285 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:36:22.796293 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 04:36:22.796373 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:36:22.796446 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:36:22.796563 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.792915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54019 (local address 127.0.98.2:38203)
0924 04:36:22.793037 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.793041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.793054 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:22.793095 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.793099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.793179 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.793250 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:22.793486 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.794223 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.795473 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.795689 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.795810 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.796092 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:22.796181 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:22.796270 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.796384 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.796429 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:36:22.818356 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:22.818521 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:22.820627 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa: Sending DeleteTablet for 3 replicas of tablet 31721a5425a147e787a767794c1094a2
I20260924 04:36:22.821233 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet 31721a5425a147e787a767794c1094a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45774
I20260924 04:36:22.821404 8444 tablet_replica.cc:330] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: stopping tablet replica
I20260924 04:36:22.821406 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet 31721a5425a147e787a767794c1094a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52532
I20260924 04:36:22.821513 8444 raft_consensus.cc:2229] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:22.821556 8442 tablet_replica.cc:330] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: stopping tablet replica
I20260924 04:36:22.821622 8444 raft_consensus.cc:2258] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:22.821652 8442 raft_consensus.cc:2229] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:22.821758 8442 raft_consensus.cc:2258] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:22.822882 8442 ts_tablet_manager.cc:1905] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:22.825397 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet 31721a5425a147e787a767794c1094a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260924 04:36:22.825598 8443 tablet_replica.cc:330] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: stopping tablet replica
I20260924 04:36:22.825757 8443 raft_consensus.cc:2229] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:22.825839 8442 ts_tablet_manager.cc:1918] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:36:22.825901 8442 log.cc:1196] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/31721a5425a147e787a767794c1094a2
I20260924 04:36:22.826058 8443 raft_consensus.cc:2258] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:22.826165 8442 ts_tablet_manager.cc:1939] T 31721a5425a147e787a767794c1094a2 P 8f03a802c8f34250b81fa5392eee1915: Deleting consensus metadata
I20260924 04:36:22.826797 8443 ts_tablet_manager.cc:1905] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:22.826958 8444 ts_tablet_manager.cc:1905] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:22.829854 8444 ts_tablet_manager.cc:1918] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:36:22.829949 8443 ts_tablet_manager.cc:1918] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:36:22.829978 8444 log.cc:1196] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/31721a5425a147e787a767794c1094a2
I20260924 04:36:22.830030 8443 log.cc:1196] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/31721a5425a147e787a767794c1094a2
I20260924 04:36:22.830171 7893 catalog_manager.cc:4841] TS 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3:45369): tablet 31721a5425a147e787a767794c1094a2 (table test-table [id=9884ac54bd54478e820296917d29379b]) successfully deleted
I20260924 04:36:22.830307 8443 ts_tablet_manager.cc:1939] T 31721a5425a147e787a767794c1094a2 P 78b6bf862ff74d2799096f4985afb3e1: Deleting consensus metadata
I20260924 04:36:22.830633 8444 ts_tablet_manager.cc:1939] T 31721a5425a147e787a767794c1094a2 P 35854587ddfb4f88ac5bca4bbc099103: Deleting consensus metadata
I20260924 04:36:22.831254 7894 catalog_manager.cc:4841] TS 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203): tablet 31721a5425a147e787a767794c1094a2 (table test-table [id=9884ac54bd54478e820296917d29379b]) successfully deleted
I20260924 04:36:22.831646 7894 catalog_manager.cc:4841] TS 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093): tablet 31721a5425a147e787a767794c1094a2 (table test-table [id=9884ac54bd54478e820296917d29379b]) successfully deleted
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:36:22.861349 7956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:22.852383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.0.98.62:45109)
0924 04:36:22.852513 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.852517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.853225 (+ 708us) server_negotiation.cc:407] Connection header received
0924 04:36:22.853453 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.853457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.853511 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.853565 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:22.854606 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.855405 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.856672 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.857049 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.857129 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.861228 (+ 4099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59464: BlockingRecv error: recv got EOF from 127.0.0.1:59464 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 04:36:22 dist-test-slave-jlnt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:36:22.874073 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.863846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.0.98.62:45109)
0924 04:36:22.863948 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.863952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.864415 (+ 463us) server_negotiation.cc:407] Connection header received
0924 04:36:22.864593 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.864596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.864645 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.864712 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:22.865570 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.866356 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.867437 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.867765 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.867830 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.870672 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:22.870676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:22.870679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:22.870704 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:22.872325 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.872792 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.872796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.872798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.872874 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:22.873165 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:22.873168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:22.873170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:22.873365 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:22.873463 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.873799 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.873912 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:36:22.878485 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:36:22.878928 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:22.883997 8167 tablet_service.cc:1467] Processing CreateTablet for tablet 057dbe266db24a7e9f3ac5b4cb52b37f (DEFAULT_TABLE table=test-table [id=b9a243e610e34bfb8693caf61ca18376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.884070 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 057dbe266db24a7e9f3ac5b4cb52b37f (DEFAULT_TABLE table=test-table [id=b9a243e610e34bfb8693caf61ca18376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.884341 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057dbe266db24a7e9f3ac5b4cb52b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.884338 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057dbe266db24a7e9f3ac5b4cb52b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.886855 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 057dbe266db24a7e9f3ac5b4cb52b37f (DEFAULT_TABLE table=test-table [id=b9a243e610e34bfb8693caf61ca18376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.887138 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057dbe266db24a7e9f3ac5b4cb52b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.887526 8419 tablet_bootstrap.cc:492] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: Bootstrap starting.
I20260924 04:36:22.888423 8419 tablet_bootstrap.cc:654] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.889423 8420 tablet_bootstrap.cc:492] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: Bootstrap starting.
I20260924 04:36:22.890015 8419 tablet_bootstrap.cc:492] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: No bootstrap required, opened a new log
I20260924 04:36:22.890090 8419 ts_tablet_manager.cc:1397] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:36:22.890204 8420 tablet_bootstrap.cc:654] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.890501 8419 raft_consensus.cc:348] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.890601 8419 raft_consensus.cc:374] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.890628 8419 raft_consensus.cc:729] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b6bf862ff74d2799096f4985afb3e1, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.890693 8418 tablet_bootstrap.cc:492] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: Bootstrap starting.
I20260924 04:36:22.890735 8419 consensus_queue.cc:260] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.890957 8419 ts_tablet_manager.cc:1428] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:36:22.891438 8418 tablet_bootstrap.cc:654] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.892759 8418 tablet_bootstrap.cc:492] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: No bootstrap required, opened a new log
I20260924 04:36:22.892850 8418 ts_tablet_manager.cc:1397] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:22.893289 8418 raft_consensus.cc:348] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.893415 8418 raft_consensus.cc:374] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.893461 8418 raft_consensus.cc:729] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f03a802c8f34250b81fa5392eee1915, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.893574 8418 consensus_queue.cc:260] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.893852 8418 ts_tablet_manager.cc:1428] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:22.895903 8420 tablet_bootstrap.cc:492] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: No bootstrap required, opened a new log
I20260924 04:36:22.896018 8420 ts_tablet_manager.cc:1397] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 04:36:22.896378 8420 raft_consensus.cc:348] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.896534 8420 raft_consensus.cc:374] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.896571 8420 raft_consensus.cc:729] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35854587ddfb4f88ac5bca4bbc099103, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.896699 8420 consensus_queue.cc:260] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.896955 8420 ts_tablet_manager.cc:1428] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:36:22.902387 8468 raft_consensus.cc:482] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:22.902506 8468 raft_consensus.cc:504] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.902882 8468 leader_election.cc:288] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093), 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203)
I20260924 04:36:22.903179 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057dbe266db24a7e9f3ac5b4cb52b37f" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35854587ddfb4f88ac5bca4bbc099103" is_pre_election: true
I20260924 04:36:22.903302 8064 raft_consensus.cc:2454] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 0.
I20260924 04:36:22.903373 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057dbe266db24a7e9f3ac5b4cb52b37f" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6bf862ff74d2799096f4985afb3e1" is_pre_election: true
I20260924 04:36:22.903506 8204 raft_consensus.cc:2454] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 0.
I20260924 04:36:22.903585 8278 leader_election.cc:302] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35854587ddfb4f88ac5bca4bbc099103, 8f03a802c8f34250b81fa5392eee1915; no voters:
I20260924 04:36:22.903755 8468 raft_consensus.cc:2781] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:22.903812 8468 raft_consensus.cc:482] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:22.903851 8468 raft_consensus.cc:3037] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.904978 8468 raft_consensus.cc:504] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.905355 8468 leader_election.cc:288] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 election: Requested vote from peers 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093), 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203)
I20260924 04:36:22.905673 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057dbe266db24a7e9f3ac5b4cb52b37f" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35854587ddfb4f88ac5bca4bbc099103"
I20260924 04:36:22.905753 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057dbe266db24a7e9f3ac5b4cb52b37f" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6bf862ff74d2799096f4985afb3e1"
I20260924 04:36:22.905788 8064 raft_consensus.cc:3037] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.905853 8204 raft_consensus.cc:3037] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:22.906874 8064 raft_consensus.cc:2454] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 1.
I20260924 04:36:22.906890 8204 raft_consensus.cc:2454] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 1.
I20260924 04:36:22.907140 8278 leader_election.cc:302] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35854587ddfb4f88ac5bca4bbc099103, 8f03a802c8f34250b81fa5392eee1915; no voters:
I20260924 04:36:22.907332 8468 raft_consensus.cc:2781] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:22.907424 8468 raft_consensus.cc:686] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 1 LEADER]: Becoming Leader. State: Replica: 8f03a802c8f34250b81fa5392eee1915, State: Running, Role: LEADER
I20260924 04:36:22.907544 8468 consensus_queue.cc:237] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.908946 7905 catalog_manager.cc:5495] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } health_report { overall_health: HEALTHY } } }
I20260924 04:36:22.918799 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:22.915225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.0.98.3:45369)
0924 04:36:22.915332 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:22.915335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:22.915376 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:36:22.915509 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:22.915512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:22.915552 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:22.915631 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:22.915909 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.916419 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.917720 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:22.917925 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:22.918002 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:22.918285 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:22.918352 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:22.918408 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:22.918522 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:36:22.918632 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:36:22.920934 8204 raft_consensus.cc:1261] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Refusing update from remote peer 8f03a802c8f34250b81fa5392eee1915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:22.920961 8064 raft_consensus.cc:1261] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Refusing update from remote peer 8f03a802c8f34250b81fa5392eee1915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:22.921411 8468 consensus_queue.cc:1035] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:22.921540 8461 consensus_queue.cc:1035] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:22.931594 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:22.931723 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:22.934504 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa: Sending DeleteTablet for 3 replicas of tablet 057dbe266db24a7e9f3ac5b4cb52b37f
I20260924 04:36:22.934916 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet 057dbe266db24a7e9f3ac5b4cb52b37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45774
I20260924 04:36:22.935066 8444 tablet_replica.cc:330] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: stopping tablet replica
I20260924 04:36:22.935180 8444 raft_consensus.cc:2229] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:22.935297 8444 raft_consensus.cc:2258] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:22.936232 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet 057dbe266db24a7e9f3ac5b4cb52b37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260924 04:36:22.936407 8443 tablet_replica.cc:330] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: stopping tablet replica
I20260924 04:36:22.936524 8443 raft_consensus.cc:2229] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:22.936633 8443 raft_consensus.cc:2258] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:22.936740 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet 057dbe266db24a7e9f3ac5b4cb52b37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260924 04:36:22.937273 8443 ts_tablet_manager.cc:1905] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:22.937284 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet 057dbe266db24a7e9f3ac5b4cb52b37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52532
I20260924 04:36:22.937426 8442 tablet_replica.cc:330] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: stopping tablet replica
I20260924 04:36:22.937515 8442 raft_consensus.cc:2229] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:22.937786 8442 raft_consensus.cc:2258] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:22.938931 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:36:22.939343 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:22.939589 8443 ts_tablet_manager.cc:1918] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:22.939652 8443 log.cc:1196] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/057dbe266db24a7e9f3ac5b4cb52b37f
I20260924 04:36:22.939890 8443 ts_tablet_manager.cc:1939] T 057dbe266db24a7e9f3ac5b4cb52b37f P 78b6bf862ff74d2799096f4985afb3e1: Deleting consensus metadata
I20260924 04:36:22.940706 8444 ts_tablet_manager.cc:1905] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:22.943095 8444 ts_tablet_manager.cc:1918] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:22.943179 8444 log.cc:1196] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/057dbe266db24a7e9f3ac5b4cb52b37f
I20260924 04:36:22.943418 8444 ts_tablet_manager.cc:1939] T 057dbe266db24a7e9f3ac5b4cb52b37f P 35854587ddfb4f88ac5bca4bbc099103: Deleting consensus metadata
I20260924 04:36:22.944392 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 09ebd278c77740f08628aa2636e7948a (DEFAULT_TABLE table=test-table [id=a4c0acd9e3754435a3d4422b865f0391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.944618 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ebd278c77740f08628aa2636e7948a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.945518 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 09ebd278c77740f08628aa2636e7948a (DEFAULT_TABLE table=test-table [id=a4c0acd9e3754435a3d4422b865f0391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.945752 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ebd278c77740f08628aa2636e7948a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.948061 8418 tablet_bootstrap.cc:492] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: Bootstrap starting.
I20260924 04:36:22.948827 8418 tablet_bootstrap.cc:654] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.949407 8167 tablet_service.cc:1467] Processing CreateTablet for tablet 09ebd278c77740f08628aa2636e7948a (DEFAULT_TABLE table=test-table [id=a4c0acd9e3754435a3d4422b865f0391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:22.949661 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ebd278c77740f08628aa2636e7948a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:22.950832 8420 tablet_bootstrap.cc:492] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103: Bootstrap starting.
I20260924 04:36:22.951563 8420 tablet_bootstrap.cc:654] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.951967 8419 tablet_bootstrap.cc:492] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: Bootstrap starting.
I20260924 04:36:22.952435 7894 catalog_manager.cc:4841] TS 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203): tablet 057dbe266db24a7e9f3ac5b4cb52b37f (table test-table [id=b9a243e610e34bfb8693caf61ca18376]) successfully deleted
W20260924 04:36:22.952669 7894 catalog_manager.cc:4808] TS 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203): delete failed for tablet 057dbe266db24a7e9f3ac5b4cb52b37f because the tablet was not found. No further retry: Not found: Tablet not found: 057dbe266db24a7e9f3ac5b4cb52b37f
I20260924 04:36:22.952813 8419 tablet_bootstrap.cc:654] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:22.953377 7894 catalog_manager.cc:4841] TS 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093): tablet 057dbe266db24a7e9f3ac5b4cb52b37f (table test-table [id=b9a243e610e34bfb8693caf61ca18376]) successfully deleted
I20260924 04:36:22.954161 8442 ts_tablet_manager.cc:1905] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:22.954432 8418 tablet_bootstrap.cc:492] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: No bootstrap required, opened a new log
I20260924 04:36:22.954506 8418 ts_tablet_manager.cc:1397] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 04:36:22.954900 8418 raft_consensus.cc:348] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.955005 8418 raft_consensus.cc:374] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.955035 8418 raft_consensus.cc:729] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f03a802c8f34250b81fa5392eee1915, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.955140 8418 consensus_queue.cc:260] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.955391 8418 ts_tablet_manager.cc:1428] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:22.956274 8442 ts_tablet_manager.cc:1918] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:22.956331 8442 log.cc:1196] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/057dbe266db24a7e9f3ac5b4cb52b37f
I20260924 04:36:22.956562 8442 ts_tablet_manager.cc:1939] T 057dbe266db24a7e9f3ac5b4cb52b37f P 8f03a802c8f34250b81fa5392eee1915: Deleting consensus metadata
I20260924 04:36:22.957701 7893 catalog_manager.cc:4841] TS 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3:45369): tablet 057dbe266db24a7e9f3ac5b4cb52b37f (table test-table [id=b9a243e610e34bfb8693caf61ca18376]) successfully deleted
I20260924 04:36:22.958092 8419 tablet_bootstrap.cc:492] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: No bootstrap required, opened a new log
I20260924 04:36:22.958186 8419 ts_tablet_manager.cc:1397] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 04:36:22.958624 8419 raft_consensus.cc:348] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.958717 8420 tablet_bootstrap.cc:492] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103: No bootstrap required, opened a new log
I20260924 04:36:22.958734 8419 raft_consensus.cc:374] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.958765 8419 raft_consensus.cc:729] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b6bf862ff74d2799096f4985afb3e1, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.958791 8420 ts_tablet_manager.cc:1397] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260924 04:36:22.958884 8419 consensus_queue.cc:260] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.959146 8420 raft_consensus.cc:348] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.959241 8420 raft_consensus.cc:374] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:22.959270 8420 raft_consensus.cc:729] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35854587ddfb4f88ac5bca4bbc099103, State: Initialized, Role: FOLLOWER
I20260924 04:36:22.959374 8420 consensus_queue.cc:260] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:22.959463 8419 ts_tablet_manager.cc:1428] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:22.959604 8420 ts_tablet_manager.cc:1428] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:36:23.178973 8469 raft_consensus.cc:482] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:23.179147 8469 raft_consensus.cc:504] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:23.179728 8469 leader_election.cc:288] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093), 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203)
I20260924 04:36:23.181293 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ebd278c77740f08628aa2636e7948a" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35854587ddfb4f88ac5bca4bbc099103" is_pre_election: true
I20260924 04:36:23.181447 8064 raft_consensus.cc:2454] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 0.
I20260924 04:36:23.181459 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ebd278c77740f08628aa2636e7948a" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6bf862ff74d2799096f4985afb3e1" is_pre_election: true
I20260924 04:36:23.181631 8204 raft_consensus.cc:2454] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 0.
I20260924 04:36:23.181775 8278 leader_election.cc:302] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35854587ddfb4f88ac5bca4bbc099103, 8f03a802c8f34250b81fa5392eee1915; no voters:
I20260924 04:36:23.182049 8469 raft_consensus.cc:2781] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:23.182102 8469 raft_consensus.cc:482] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:23.182143 8469 raft_consensus.cc:3037] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:23.183311 8469 raft_consensus.cc:504] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:23.183622 8469 leader_election.cc:288] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 election: Requested vote from peers 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093), 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203)
I20260924 04:36:23.183983 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ebd278c77740f08628aa2636e7948a" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35854587ddfb4f88ac5bca4bbc099103"
I20260924 04:36:23.184041 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ebd278c77740f08628aa2636e7948a" candidate_uuid: "8f03a802c8f34250b81fa5392eee1915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b6bf862ff74d2799096f4985afb3e1"
I20260924 04:36:23.184098 8064 raft_consensus.cc:3037] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:23.184150 8204 raft_consensus.cc:3037] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:23.185211 8064 raft_consensus.cc:2454] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 1.
I20260924 04:36:23.185211 8204 raft_consensus.cc:2454] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f03a802c8f34250b81fa5392eee1915 in term 1.
I20260924 04:36:23.185525 8278 leader_election.cc:302] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b6bf862ff74d2799096f4985afb3e1, 8f03a802c8f34250b81fa5392eee1915; no voters:
I20260924 04:36:23.185756 8469 raft_consensus.cc:2781] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:23.185833 8469 raft_consensus.cc:686] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 1 LEADER]: Becoming Leader. State: Replica: 8f03a802c8f34250b81fa5392eee1915, State: Running, Role: LEADER
I20260924 04:36:23.185952 8469 consensus_queue.cc:237] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } }
I20260924 04:36:23.187528 7906 catalog_manager.cc:5495] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8f03a802c8f34250b81fa5392eee1915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f03a802c8f34250b81fa5392eee1915" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45369 } health_report { overall_health: HEALTHY } } }
I20260924 04:36:23.258176 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:23.255030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45369 (local address 127.0.98.3:52611)
0924 04:36:23.255191 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:23.255207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:23.255295 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:23.255517 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:23.255520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:23.255526 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:36:23.255700 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:23.255706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:23.256419 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:23.256421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:23.257095 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:23.257105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:23.257610 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:23.257612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:23.257671 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:23.257694 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:36:23.257882 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:36:23.257886 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:36:23.257934 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:36:23.258051 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":22}
I20260924 04:36:23.258219 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:23.255108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52611 (local address 127.0.98.3:45369)
0924 04:36:23.255208 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:23.255212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:23.255251 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:36:23.255391 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:23.255394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:23.255438 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:23.255512 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:23.255795 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:23.256289 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:23.257224 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:23.257491 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:23.257571 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:23.257767 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:23.257822 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:23.257868 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:23.257978 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:36:23.258050 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":40}
I20260924 04:36:23.262802 8204 raft_consensus.cc:1261] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Refusing update from remote peer 8f03a802c8f34250b81fa5392eee1915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:23.262822 8064 raft_consensus.cc:1261] T 09ebd278c77740f08628aa2636e7948a P 35854587ddfb4f88ac5bca4bbc099103 [term 1 FOLLOWER]: Refusing update from remote peer 8f03a802c8f34250b81fa5392eee1915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:23.263170 8469 consensus_queue.cc:1035] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35854587ddfb4f88ac5bca4bbc099103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:23.263378 8424 consensus_queue.cc:1035] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b6bf862ff74d2799096f4985afb3e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:23.292199 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:23.292331 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:23.294091 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc55a982086f481c88a40924e13ea2fa: Sending DeleteTablet for 3 replicas of tablet 09ebd278c77740f08628aa2636e7948a
I20260924 04:36:23.294596 8167 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ebd278c77740f08628aa2636e7948a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44868
I20260924 04:36:23.294761 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ebd278c77740f08628aa2636e7948a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45774
I20260924 04:36:23.294914 8443 tablet_replica.cc:330] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: stopping tablet replica
I20260924 04:36:23.295024 8443 raft_consensus.cc:2229] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:23.295116 8443 raft_consensus.cc:2258] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:23.295145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 7979
I20260924 04:36:23.296499 8443 ts_tablet_manager.cc:1905] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:23.297099 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ebd278c77740f08628aa2636e7948a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52532
I20260924 04:36:23.297319 8442 tablet_replica.cc:330] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: stopping tablet replica
I20260924 04:36:23.297464 8442 raft_consensus.cc:2229] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:23.297649 8442 raft_consensus.cc:2258] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:23.298344 8442 ts_tablet_manager.cc:1905] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:23.299185 8443 ts_tablet_manager.cc:1918] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:36:23.299260 8443 log.cc:1196] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/09ebd278c77740f08628aa2636e7948a
I20260924 04:36:23.299494 8443 ts_tablet_manager.cc:1939] T 09ebd278c77740f08628aa2636e7948a P 78b6bf862ff74d2799096f4985afb3e1: Deleting consensus metadata
I20260924 04:36:23.300112 7894 catalog_manager.cc:4841] TS 78b6bf862ff74d2799096f4985afb3e1 (127.0.98.2:38203): tablet 09ebd278c77740f08628aa2636e7948a (table test-table [id=a4c0acd9e3754435a3d4422b865f0391]) successfully deleted
I20260924 04:36:23.300603 8442 ts_tablet_manager.cc:1918] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:36:23.300704 8442 log.cc:1196] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/09ebd278c77740f08628aa2636e7948a
I20260924 04:36:23.300936 8442 ts_tablet_manager.cc:1939] T 09ebd278c77740f08628aa2636e7948a P 8f03a802c8f34250b81fa5392eee1915: Deleting consensus metadata
I20260924 04:36:23.301548 7893 catalog_manager.cc:4841] TS 8f03a802c8f34250b81fa5392eee1915 (127.0.98.3:45369): tablet 09ebd278c77740f08628aa2636e7948a (table test-table [id=a4c0acd9e3754435a3d4422b865f0391]) successfully deleted
W20260924 04:36:23.306573 7894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38093 (error 108)
I20260924 04:36:23.306879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 8119
I20260924 04:36:23.308851 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:23.308429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38093 (local address 127.0.0.1:45780)
0924 04:36:23.308642 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:23.308732 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
W20260924 04:36:23.308964 7894 catalog_manager.cc:4584] TS 35854587ddfb4f88ac5bca4bbc099103 (127.0.98.1:38093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09ebd278c77740f08628aa2636e7948a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38093: connect: Connection refused (error 111)
I20260924 04:36:23.314349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 8259
I20260924 04:36:23.321751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 7875
2026-09-24T04:36:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3077 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:23 dist-test-slave-jlnt krb5kdc[8510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:23 dist-test-slave-jlnt krb5kdc[8510](info): set up 2 sockets
Sep 24 04:36:23 dist-test-slave-jlnt krb5kdc[8510](info): commencing operation
krb5kdc: starting...
W20260924 04:36:24.353859 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:24 dist-test-slave-jlnt krb5kdc[8510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:36:24.471398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33935
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:24.592060 8523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:24.592304 8523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:24.592344 8523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:24.596274 8523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:24.596328 8523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:24.596347 8523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:24.596364 8523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:24.596379 8523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:24.601456 8523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:24.602336 8523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:24.603241 8523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:24.611325 8528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:24.611343 8529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:24.611421 8523 server_base.cc:1034] running on GCE node
W20260924 04:36:24.611325 8531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:24.612150 8523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:24.613281 8523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:24.614449 8523 hybrid_clock.cc:648] HybridClock initialized: now 1790224584614429 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T04:36:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1295 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:24 dist-test-slave-jlnt krb5kdc[8536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:24 dist-test-slave-jlnt krb5kdc[8536](info): set up 2 sockets
Sep 24 04:36:24 dist-test-slave-jlnt krb5kdc[8536](info): commencing operation
krb5kdc: starting...
W20260924 04:36:25.669850 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:25 dist-test-slave-jlnt krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:36:25.788368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:25.895085 8549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:25.895354 8549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:25.895394 8549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:25.899266 8549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:25.899318 8549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:25.899338 8549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:25.899353 8549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:25.899369 8549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:25.904481 8549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:25.905457 8549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:25.906405 8549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:25.914599 8557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:25.914605 8555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:25.914665 8554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:25.914858 8549 server_base.cc:1034] running on GCE node
I20260924 04:36:25.915478 8549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:25.916514 8549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:25.917689 8549 hybrid_clock.cc:648] HybridClock initialized: now 1790224585917677 us; error 27 us; skew 500 ppm
Sep 24 04:36:25 dist-test-slave-jlnt krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:25.920336 8549 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:36:25.921310 8549 webserver.cc:460] Webserver started at http://127.0.98.62:42619/ using document root <none> and password file <none>
I20260924 04:36:25.921855 8549 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:25.921916 8549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:25.922148 8549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:25.924155 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "e52aecf1439f45518ed21c9b88649975"
format_stamp: "Formatted at 2026-09-24 04:36:25 on dist-test-slave-jlnt"
server_key: "b2ba1a23a8be0cb596aaac57c683725e"
server_key_iv: "bda5a7705092dc11e178def4741a916b"
server_key_version: "encryptionkey@0"
I20260924 04:36:25.924732 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "e52aecf1439f45518ed21c9b88649975"
format_stamp: "Formatted at 2026-09-24 04:36:25 on dist-test-slave-jlnt"
server_key: "b2ba1a23a8be0cb596aaac57c683725e"
server_key_iv: "bda5a7705092dc11e178def4741a916b"
server_key_version: "encryptionkey@0"
I20260924 04:36:25.928395 8549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:36:25.930789 8564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:25.931988 8549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:25.932124 8549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "e52aecf1439f45518ed21c9b88649975"
format_stamp: "Formatted at 2026-09-24 04:36:25 on dist-test-slave-jlnt"
server_key: "b2ba1a23a8be0cb596aaac57c683725e"
server_key_iv: "bda5a7705092dc11e178def4741a916b"
server_key_version: "encryptionkey@0"
I20260924 04:36:25.932233 8549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:25.949299 8549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:25.952034 8549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:25.952226 8549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:25.959178 8549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34689
I20260924 04:36:25.959192 8616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34689 every 8 connection(s)
I20260924 04:36:25.960084 8549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:25.962803 8617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:25.963953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 8549
I20260924 04:36:25.964098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:25.964430 392 external_mini_cluster.cc:1428] Setting key 989030098294269fbc80867deca95874
I20260924 04:36:25.969085 8617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: Bootstrap starting.
I20260924 04:36:25.971684 8617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:25.972601 8617 log.cc:824] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:36:25 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:25.974618 8617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: No bootstrap required, opened a new log
I20260924 04:36:25.978519 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:25.965795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.0.98.62:34689)
0924 04:36:25.966177 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:25.966202 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:25.966237 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:36:25.966852 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:25.966867 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:25.967200 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:25.967471 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:25.968335 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:25.969405 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:25.970598 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:25.971058 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:25.971174 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:25.973934 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:25.973939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:25.973952 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:25.973977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:25.975709 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:25.976196 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:25.976201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:25.976205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:25.976309 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:25.976645 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:25.976648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:25.976649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:25.977039 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:25.977278 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:25.977514 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 04:36:25.977823 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":124,"threads_started":1}
I20260924 04:36:25.978432 8617 raft_consensus.cc:348] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260924 04:36:25.978684 8617 raft_consensus.cc:374] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:25.978772 8617 raft_consensus.cc:729] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52aecf1439f45518ed21c9b88649975, State: Initialized, Role: FOLLOWER
I20260924 04:36:25.979251 8617 consensus_queue.cc:260] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260924 04:36:25.979403 8617 raft_consensus.cc:388] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:25.979465 8617 raft_consensus.cc:482] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:25.979554 8617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:25.980417 8617 raft_consensus.cc:504] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260924 04:36:25.980782 8617 leader_election.cc:302] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e52aecf1439f45518ed21c9b88649975; no voters:
I20260924 04:36:25.981082 8617 leader_election.cc:288] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:25.981218 8622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:25.981454 8622 raft_consensus.cc:686] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 1 LEADER]: Becoming Leader. State: Replica: e52aecf1439f45518ed21c9b88649975, State: Running, Role: LEADER
I20260924 04:36:25.981806 8622 consensus_queue.cc:237] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260924 04:36:25.982273 8617 sys_catalog.cc:564] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:25.983415 8624 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e52aecf1439f45518ed21c9b88649975. Latest consensus state: current_term: 1 leader_uuid: "e52aecf1439f45518ed21c9b88649975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } } }
I20260924 04:36:25.983557 8624 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:25.983807 8623 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e52aecf1439f45518ed21c9b88649975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } } }
I20260924 04:36:25.983887 8631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:25.983942 8623 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:25.987025 8631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:25.992641 8631 catalog_manager.cc:1347] Generated new cluster ID: cceb51b7d8354f548f0ba063fa111159
I20260924 04:36:25.992714 8631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:25.998698 8631 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:25.999532 8631 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:26.006536 8631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: Generated new TSK 0
I20260924 04:36:26.007289 8631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:36:26.048447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34689
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:26.157032 8643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:26.157296 8643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:26.157366 8643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:26.161327 8643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:26.161377 8643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:26.161434 8643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:26.166579 8643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:26.167568 8643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:26.168571 8643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:26.178251 8649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:26.178278 8651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:26.178318 8648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:26.178598 8643 server_base.cc:1034] running on GCE node
I20260924 04:36:26.178979 8643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:26.179616 8643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:26.180812 8643 hybrid_clock.cc:648] HybridClock initialized: now 1790224586180783 us; error 39 us; skew 500 ppm
Sep 24 04:36:26 dist-test-slave-jlnt krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:26.183768 8643 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:36:26.184605 8643 webserver.cc:460] Webserver started at http://127.0.98.1:35567/ using document root <none> and password file <none>
I20260924 04:36:26.185214 8643 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:26.185312 8643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:26.185519 8643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:26.187486 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "1c40c0df2b4a48a693624c66515bbf50"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "0f26242b5c710646faee11b7e4469039"
server_key_iv: "0e6e227ab588f82196f1e62db3f1e088"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.188055 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1c40c0df2b4a48a693624c66515bbf50"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "0f26242b5c710646faee11b7e4469039"
server_key_iv: "0e6e227ab588f82196f1e62db3f1e088"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.191682 8643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:26.194109 8658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.195326 8643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:26.195470 8643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "1c40c0df2b4a48a693624c66515bbf50"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "0f26242b5c710646faee11b7e4469039"
server_key_iv: "0e6e227ab588f82196f1e62db3f1e088"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.195564 8643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:26.230888 8643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:26.234021 8643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:26.234233 8643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:26.234824 8643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:26.235675 8643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:26.235759 8643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.235854 8643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:26.235905 8643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.244441 8643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44645
I20260924 04:36:26.244474 8771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44645 every 8 connection(s)
I20260924 04:36:26.245339 8643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:36:26 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:26.254920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 8643
I20260924 04:36:26.255051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:26.255352 392 external_mini_cluster.cc:1428] Setting key 250c0e01765b2c6cd0c43b9dce6cba13
I20260924 04:36:26.259456 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:26.247039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42329 (local address 127.0.98.62:34689)
0924 04:36:26.247177 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:26.247180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:26.247976 (+ 796us) server_negotiation.cc:407] Connection header received
0924 04:36:26.249378 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:26.249381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:26.249438 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:26.249548 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:26.250638 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.251182 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.252324 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.252563 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.252678 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:26.255222 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:26.255225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:26.255227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:26.255250 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:26.257062 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:26.257690 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:26.257693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:26.257694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:26.257770 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:26.258221 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:26.258223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:26.258224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:26.258386 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:26.258459 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:26.259101 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 04:36:26.259250 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:36:26.259979 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:26.247419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:42329)
0924 04:36:26.247838 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:26.247878 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:26.249126 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:26.249718 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:26.249725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:26.250057 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:26.250495 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:26.250507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.251328 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.251333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:26.252199 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:26.252207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.252722 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.252726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:26.252884 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:26.253503 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:26.253518 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:26.255025 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:26.257192 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:26.257199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:26.257215 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:26.257568 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:26.257900 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:26.257906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:26.257911 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:26.258102 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:26.258486 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:26.258494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:26.258825 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 04:36:26.258975 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 04:36:26.259211 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20260924 04:36:26.261186 8772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34689
I20260924 04:36:26.261399 8772 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:26.261852 8772 heartbeater.cc:511] Master 127.0.98.62:34689 requested a full tablet report, sending...
I20260924 04:36:26.263247 8581 ts_manager.cc:194] Registered new tserver with Master: 1c40c0df2b4a48a693624c66515bbf50 (127.0.98.1:44645)
I20260924 04:36:26.264335 8581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42329
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:36:26.282480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34689
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:26.388223 8777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:26.388489 8777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:26.388561 8777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:26.392531 8777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:26.392617 8777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:26.392704 8777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:26.397733 8777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:26.398712 8777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:26.399737 8777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:26.408784 8782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:26.408790 8785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:26.408790 8783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:26.409150 8777 server_base.cc:1034] running on GCE node
I20260924 04:36:26.409539 8777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:26.410120 8777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:26.411296 8777 hybrid_clock.cc:648] HybridClock initialized: now 1790224586411276 us; error 33 us; skew 500 ppm
Sep 24 04:36:26 dist-test-slave-jlnt krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:26.413851 8777 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:36:26.414626 8777 webserver.cc:460] Webserver started at http://127.0.98.2:40939/ using document root <none> and password file <none>
I20260924 04:36:26.415148 8777 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:26.415232 8777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:26.415450 8777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:26.417413 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "39347934744945e180da6bbc1a58a474"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "fd9951ad64f64e78926dba2055f15a0f"
server_key_iv: "22b228a0a06c1c26b98fc61560155541"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.417954 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "39347934744945e180da6bbc1a58a474"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "fd9951ad64f64e78926dba2055f15a0f"
server_key_iv: "22b228a0a06c1c26b98fc61560155541"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.421456 8777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:36:26.423710 8792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.424960 8777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:26.425115 8777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "39347934744945e180da6bbc1a58a474"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "fd9951ad64f64e78926dba2055f15a0f"
server_key_iv: "22b228a0a06c1c26b98fc61560155541"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.425271 8777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:26.435353 8777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:26.438274 8777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:26.438457 8777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:26.438969 8777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:26.439769 8777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:26.439837 8777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.439904 8777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:26.439950 8777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.448758 8777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35915
I20260924 04:36:26.448783 8905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35915 every 8 connection(s)
I20260924 04:36:26.449653 8777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:26.457885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 8777
I20260924 04:36:26.458043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:26.458325 392 external_mini_cluster.cc:1428] Setting key d7b37b874edc6452b847900a7fdb7025
Sep 24 04:36:26 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:26.463976 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:26.451441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37485 (local address 127.0.98.62:34689)
0924 04:36:26.451621 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:26.451624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:26.452354 (+ 730us) server_negotiation.cc:407] Connection header received
0924 04:36:26.453776 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:26.453778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:26.453825 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:26.453927 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:26.455102 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.455639 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.456780 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.457018 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.457122 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:26.459980 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:26.459983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:26.459986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:26.460010 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:26.461840 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:26.462376 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:26.462383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:26.462387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:26.462473 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:26.462818 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:26.462823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:26.462825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:26.463012 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:26.463123 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:26.463684 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 04:36:26.463817 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 04:36:26.464855 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:26.451766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:37485)
0924 04:36:26.452194 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:26.452234 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:26.453558 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:26.454114 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:26.454121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:26.454462 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:26.454940 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:26.454954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.455763 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.455766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:26.456640 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:26.456648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.457234 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.457243 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:26.457382 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:26.458043 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:26.458065 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:26.459803 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:26.461964 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:26.461970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:26.461986 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:26.462266 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:26.462571 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:26.462574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:26.462576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:26.462719 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:26.463121 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:26.463128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:26.463390 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 04:36:26.463645 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 04:36:26.464004 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260924 04:36:26.466074 8906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34689
I20260924 04:36:26.466331 8906 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:26.466823 8906 heartbeater.cc:511] Master 127.0.98.62:34689 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:36:26.467880 8581 ts_manager.cc:194] Registered new tserver with Master: 39347934744945e180da6bbc1a58a474 (127.0.98.2:35915)
I20260924 04:36:26.468509 8581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:36:26.484025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34689
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:26.592770 8911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:26.593003 8911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:26.593045 8911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:26.597000 8911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:26.597054 8911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:26.597111 8911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:26.602293 8911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:26.603199 8911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:26.604105 8911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:26.613186 8917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:26.613193 8916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:26.613207 8919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:26.613428 8911 server_base.cc:1034] running on GCE node
I20260924 04:36:26.613858 8911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:26.614491 8911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:26.615674 8911 hybrid_clock.cc:648] HybridClock initialized: now 1790224586615639 us; error 48 us; skew 500 ppm
Sep 24 04:36:26 dist-test-slave-jlnt krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:26.626729 8911 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:36:26.627622 8911 webserver.cc:460] Webserver started at http://127.0.98.3:38359/ using document root <none> and password file <none>
I20260924 04:36:26.628197 8911 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:26.628286 8911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:26.628508 8911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:26.630627 8911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "bde42fe7ac6245c2ba6574998d41052e"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "447fc1759b072949a835b38ba452b3ce"
server_key_iv: "9ad5ace153f79ba0d92a82e7ec11a502"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.631191 8911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bde42fe7ac6245c2ba6574998d41052e"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "447fc1759b072949a835b38ba452b3ce"
server_key_iv: "9ad5ace153f79ba0d92a82e7ec11a502"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.634785 8911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:36:26.637255 8926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.638398 8911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:26.638525 8911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "bde42fe7ac6245c2ba6574998d41052e"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "447fc1759b072949a835b38ba452b3ce"
server_key_iv: "9ad5ace153f79ba0d92a82e7ec11a502"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.638628 8911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:26.655711 8911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:26.658607 8911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:26.658805 8911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:26.659391 8911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:26.660204 8911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:26.660274 8911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.660336 8911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:26.660386 8911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.669047 8911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36193
I20260924 04:36:26.669078 9039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36193 every 8 connection(s)
I20260924 04:36:26.669979 8911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:36:26 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:26.679811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 8911
I20260924 04:36:26.679966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:26.680274 392 external_mini_cluster.cc:1428] Setting key 6e55eb5fb12d0363821f99a18e7899e4
I20260924 04:36:26.683252 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:26.671690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54065 (local address 127.0.98.62:34689)
0924 04:36:26.671827 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:26.671831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:26.672616 (+ 785us) server_negotiation.cc:407] Connection header received
0924 04:36:26.674069 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:26.674073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:26.674129 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:26.674235 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:26.675326 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.675856 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.676966 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.677251 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.677362 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:26.679631 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:26.679635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:26.679638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:26.679669 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:26.681174 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:26.681688 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:26.681693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:26.681694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:26.681773 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:26.682189 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:26.682192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:26.682193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:26.682426 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:26.682505 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:26.683040 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 04:36:26.683131 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:36:26.684183 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:26.672039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:54065)
0924 04:36:26.672472 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:26.672513 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:26.673863 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:26.674398 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:26.674404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:26.674739 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:26.675187 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:26.675199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.675977 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.675983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:26.676843 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:26.676851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:26.677393 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:26.677397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:26.677568 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:26.678096 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:26.678112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:26.679464 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:26.681306 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:26.681311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:26.681325 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:26.681581 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:26.681916 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:26.681919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:26.681921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:26.682099 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:26.682518 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:26.682523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:26.682763 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 04:36:26.682995 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 04:36:26.683295 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260924 04:36:26.685101 9040 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34689
I20260924 04:36:26.685365 9040 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:26.685814 9040 heartbeater.cc:511] Master 127.0.98.62:34689 requested a full tablet report, sending...
I20260924 04:36:26.686734 8581 ts_manager.cc:194] Registered new tserver with Master: bde42fe7ac6245c2ba6574998d41052e (127.0.98.3:36193)
I20260924 04:36:26.687230 8581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54065
I20260924 04:36:26.694002 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:36:26.765955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 8643
I20260924 04:36:26.771263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 8777
I20260924 04:36:26.776516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 8911
I20260924 04:36:26.781514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 8549
I20260924 04:36:26.787461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34689
--webserver_interface=127.0.98.62
--webserver_port=42619
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:26.892939 9049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:26.893221 9049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:26.893307 9049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:26.897267 9049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:26.897324 9049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:26.897343 9049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:26.897388 9049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:26.897424 9049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:26.902558 9049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:26.903605 9049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:26.904640 9049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:26.912987 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:26.913172 9049 server_base.cc:1034] running on GCE node
W20260924 04:36:26.912981 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:26.913384 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:26.913955 9049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:26.915004 9049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:26.916198 9049 hybrid_clock.cc:648] HybridClock initialized: now 1790224586916177 us; error 33 us; skew 500 ppm
Sep 24 04:36:26 dist-test-slave-jlnt krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:26.918926 9049 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:36:26.919781 9049 webserver.cc:460] Webserver started at http://127.0.98.62:42619/ using document root <none> and password file <none>
I20260924 04:36:26.920305 9049 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:26.920377 9049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:26.924074 9049 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:26.926190 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:26.927372 9049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:26.927502 9049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "e52aecf1439f45518ed21c9b88649975"
format_stamp: "Formatted at 2026-09-24 04:36:25 on dist-test-slave-jlnt"
server_key: "b2ba1a23a8be0cb596aaac57c683725e"
server_key_iv: "bda5a7705092dc11e178def4741a916b"
server_key_version: "encryptionkey@0"
I20260924 04:36:26.928018 9049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:26.946949 9049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:26.949797 9049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:26.950004 9049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:26.956794 9049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34689
I20260924 04:36:26.956825 9116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34689 every 8 connection(s)
I20260924 04:36:26.957799 9049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:26.961751 9117 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:36:26.962667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 9049
I20260924 04:36:26.963099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44645
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34689
--webserver_port=35567
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:36:26.963528 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: Bootstrap starting.
I20260924 04:36:26.979274 9117 log.cc:824] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:26.985572 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:36:26.985927 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: Bootstrap complete.
I20260924 04:36:26.990069 9117 raft_consensus.cc:348] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260924 04:36:26.990486 9117 raft_consensus.cc:729] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52aecf1439f45518ed21c9b88649975, State: Initialized, Role: FOLLOWER
I20260924 04:36:26.991004 9117 consensus_queue.cc:260] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260924 04:36:26.991158 9117 raft_consensus.cc:388] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:26.991245 9117 raft_consensus.cc:482] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:26.991359 9117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:36:26.993310 9117 raft_consensus.cc:504] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260924 04:36:26.993701 9117 leader_election.cc:302] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e52aecf1439f45518ed21c9b88649975; no voters:
I20260924 04:36:26.994009 9117 leader_election.cc:288] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:36:26.994145 9121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:36:26.994395 9121 raft_consensus.cc:686] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [term 2 LEADER]: Becoming Leader. State: Replica: e52aecf1439f45518ed21c9b88649975, State: Running, Role: LEADER
I20260924 04:36:26.994778 9121 consensus_queue.cc:237] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260924 04:36:26.995172 9117 sys_catalog.cc:564] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:26.996132 9122 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e52aecf1439f45518ed21c9b88649975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } } }
I20260924 04:36:26.996251 9122 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:26.996579 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e52aecf1439f45518ed21c9b88649975. Latest consensus state: current_term: 2 leader_uuid: "e52aecf1439f45518ed21c9b88649975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52aecf1439f45518ed21c9b88649975" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } } }
I20260924 04:36:26.996752 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:26.997066 9130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:27.000080 9130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:27.000698 9130 catalog_manager.cc:1259] Loaded cluster ID: cceb51b7d8354f548f0ba063fa111159
I20260924 04:36:27.000758 9130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:27.003021 9130 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:27.003649 9130 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e52aecf1439f45518ed21c9b88649975: Loaded TSK: 0
I20260924 04:36:27.004801 9130 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:36:27.077396 9119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:27.077672 9119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:27.077754 9119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:27.081858 9119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:27.081940 9119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:27.082026 9119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:27.087675 9119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35567
--tserver_master_addrs=127.0.98.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:27.088707 9119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:27.089754 9119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:27.100186 9144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:27.100313 9143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:27.100198 9146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:27.100536 9119 server_base.cc:1034] running on GCE node
I20260924 04:36:27.100931 9119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:27.101643 9119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:27.102852 9119 hybrid_clock.cc:648] HybridClock initialized: now 1790224587102826 us; error 44 us; skew 500 ppm
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224587, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:27.105615 9119 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:36:27.106498 9119 webserver.cc:460] Webserver started at http://127.0.98.1:35567/ using document root <none> and password file <none>
I20260924 04:36:27.107054 9119 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:27.107125 9119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:27.110993 9119 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 04:36:27.113078 9153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:27.114390 9119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:36:27.114524 9119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "1c40c0df2b4a48a693624c66515bbf50"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "0f26242b5c710646faee11b7e4469039"
server_key_iv: "0e6e227ab588f82196f1e62db3f1e088"
server_key_version: "encryptionkey@0"
I20260924 04:36:27.115046 9119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:27.131636 9119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:27.134894 9119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:27.135104 9119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:27.135710 9119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:27.136497 9119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:27.136566 9119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:27.136659 9119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:27.136709 9119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:27.146459 9119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44645
I20260924 04:36:27.146481 9266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44645 every 8 connection(s)
I20260924 04:36:27.147419 9119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:27.149356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 9119
I20260924 04:36:27.149852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35915
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34689
--webserver_port=40939
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:27.160847 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.149747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:37341)
0924 04:36:27.150702 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:27.150741 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:27.152041 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:27.153117 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:27.153124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:27.153563 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:27.154000 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.154013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.155273 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.155281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.156127 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.156136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.156855 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.156858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.157042 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.157673 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:27.157688 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:27.159943 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":137,"threads_started":1}
W20260924 04:36:27.160836 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:27.149631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37341 (local address 127.0.98.62:34689)
0924 04:36:27.150127 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.150148 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.150853 (+ 705us) server_negotiation.cc:407] Connection header received
0924 04:36:27.152432 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.152442 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.152727 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.153049 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.154170 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.155129 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.156253 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.156698 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.156868 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.160147 (+ 3279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37341: BlockingRecv error: recv got EOF from 127.0.98.1:37341 (error 108)
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":130,"threads_started":1}
W20260924 04:36:27.161471 9267 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34689 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34689: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:27.166991 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.161913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:53431)
0924 04:36:27.162018 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:27.162033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:27.162114 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:27.162404 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:27.162407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:27.162567 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:27.162648 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.162653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.163355 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.163358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.163885 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.163893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.164327 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.164330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.164423 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.165065 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:27.165078 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:27.166847 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:36:27.167078 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:27.161987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53431 (local address 127.0.98.62:34689)
0924 04:36:27.162229 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.162232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.162241 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:36:27.162267 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.162269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.162302 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.162391 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.162757 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.163240 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.164001 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.164219 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.164327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.166962 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53431: BlockingRecv error: recv got EOF from 127.0.98.1:53431 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:27.173296 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.167501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:47069)
0924 04:36:27.167605 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:27.167621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:27.167696 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:27.167936 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:27.167939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:27.168083 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:27.168153 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.168158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.168945 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.168951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.169584 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.169593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.170201 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.170204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.170331 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.170857 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:27.170871 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:27.173154 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:36:27.173470 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:27.167617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47069 (local address 127.0.98.62:34689)
0924 04:36:27.167747 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.167752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.167765 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:27.167804 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.167807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.167839 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.167924 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.168261 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.168845 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.169705 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.170085 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.170203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.173302 (+ 3099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47069: BlockingRecv error: recv got EOF from 127.0.98.1:47069 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 04:36:27.173724 9267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:36:27.262142 9271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:27.262424 9271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:27.262499 9271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:27.266561 9271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:27.266641 9271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:27.266732 9271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:27.271816 9271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40939
--tserver_master_addrs=127.0.98.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:27.272800 9271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:27.273806 9271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:27.282966 9279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:27.282989 9276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:27.283115 9271 server_base.cc:1034] running on GCE node
W20260924 04:36:27.282989 9277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:27.283843 9271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:27.284466 9271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:27.285619 9271 hybrid_clock.cc:648] HybridClock initialized: now 1790224587285604 us; error 26 us; skew 500 ppm
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224587, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:27.288269 9271 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:36:27.289203 9271 webserver.cc:460] Webserver started at http://127.0.98.2:40939/ using document root <none> and password file <none>
I20260924 04:36:27.289763 9271 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:27.289865 9271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:27.293596 9271 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 04:36:27.295737 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:27.297034 9271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:27.297178 9271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "39347934744945e180da6bbc1a58a474"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "fd9951ad64f64e78926dba2055f15a0f"
server_key_iv: "22b228a0a06c1c26b98fc61560155541"
server_key_version: "encryptionkey@0"
I20260924 04:36:27.297703 9271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:27.318974 9271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:27.321869 9271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:27.322021 9271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:27.322553 9271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:27.323300 9271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:27.323346 9271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:27.323385 9271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:27.323400 9271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:27.332192 9271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35915
I20260924 04:36:27.332227 9399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35915 every 8 connection(s)
I20260924 04:36:27.333240 9271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:27.336421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 9271
I20260924 04:36:27.336988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36193
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34689
--webserver_port=38359
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:36:27.345763 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:27.334900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36911 (local address 127.0.98.62:34689)
0924 04:36:27.335031 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.335035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.335864 (+ 829us) server_negotiation.cc:407] Connection header received
0924 04:36:27.337785 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.337787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.337838 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.337903 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.339221 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.340115 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.341423 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.341851 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.341989 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.345593 (+ 3604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36911: BlockingRecv error: recv got EOF from 127.0.98.2:36911 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:36:27.346217 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.335301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:36911)
0924 04:36:27.335724 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:27.335765 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:27.337535 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:27.338091 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:27.338103 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:27.338579 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:27.339060 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.339074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.340260 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.340262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.341295 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.341305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.342020 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.342023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.342180 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.342810 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:27.342825 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:27.345404 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
W20260924 04:36:27.346736 9400 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34689 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34689: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:27.352218 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.347265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:46081)
0924 04:36:27.347381 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:27.347396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:27.347474 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:27.347736 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:27.347739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:27.347898 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:27.347980 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.347984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.348699 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.348702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.349303 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.349311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.349753 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.349757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.349846 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.350346 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:27.350356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:27.352071 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 04:36:27.352293 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:27.347359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46081 (local address 127.0.98.62:34689)
0924 04:36:27.347493 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.347499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.347514 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:27.347556 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.347559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.347605 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.347723 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.348089 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.348569 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.349417 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.349639 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.349756 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.352180 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46081: BlockingRecv error: recv got EOF from 127.0.98.2:46081 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:27.357944 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.352782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:46159)
0924 04:36:27.352901 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:27.352916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:27.352987 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:27.353255 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:27.353258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:27.353433 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:27.353540 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.353549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.354221 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.354224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.355094 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.355103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.355511 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.355514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.355594 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.356023 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:27.356037 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:27.357825 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:36:27.358031 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:27.352848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46159 (local address 127.0.98.62:34689)
0924 04:36:27.352947 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.352950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.352961 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:36:27.353070 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.353072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.353103 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.353239 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.353671 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.354115 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.355215 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.355413 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.355489 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.357927 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46159: BlockingRecv error: recv got EOF from 127.0.98.2:46159 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:36:27.358295 9400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:36:27.448994 9403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:27.449290 9403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:27.449363 9403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:27.453306 9403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:27.453392 9403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:27.453482 9403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:27.458534 9403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38359
--tserver_master_addrs=127.0.98.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:27.459540 9403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:27.460531 9403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:27.469794 9411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:27.469805 9408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:27.470144 9403 server_base.cc:1034] running on GCE node
W20260924 04:36:27.469805 9409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:27.470616 9403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:27.471221 9403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:27.472399 9403 hybrid_clock.cc:648] HybridClock initialized: now 1790224587472366 us; error 46 us; skew 500 ppm
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224587, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:27.475080 9403 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:36:27.475945 9403 webserver.cc:460] Webserver started at http://127.0.98.3:38359/ using document root <none> and password file <none>
I20260924 04:36:27.476491 9403 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:27.476589 9403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:27.480423 9403 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:36:27.482573 9418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:27.483881 9403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:27.484014 9403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "bde42fe7ac6245c2ba6574998d41052e"
format_stamp: "Formatted at 2026-09-24 04:36:26 on dist-test-slave-jlnt"
server_key: "447fc1759b072949a835b38ba452b3ce"
server_key_iv: "9ad5ace153f79ba0d92a82e7ec11a502"
server_key_version: "encryptionkey@0"
I20260924 04:36:27.484529 9403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:27.505259 9403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:27.508013 9403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:27.508212 9403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:27.508751 9403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:27.509589 9403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:27.509655 9403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:27.509739 9403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:27.509785 9403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:27.518586 9403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36193
I20260924 04:36:27.518728 9531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36193 every 8 connection(s)
I20260924 04:36:27.519605 9403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:27.523468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 9403
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:36:27.531594 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:27.521360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42825 (local address 127.0.98.62:34689)
0924 04:36:27.521510 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.521514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.522362 (+ 848us) server_negotiation.cc:407] Connection header received
0924 04:36:27.523815 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.523818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.523891 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.523979 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.525445 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.526064 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.527324 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.527581 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.527711 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.531417 (+ 3706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42825: BlockingRecv error: recv got EOF from 127.0.98.3:42825 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:36:27.532485 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.521799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:42825)
0924 04:36:27.522217 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:27.522257 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:27.523579 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:27.524203 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:27.524212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:27.524650 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:27.525306 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.525321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.526194 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.526198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.527205 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.527214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.527842 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.527845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.527981 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.528464 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:27.528479 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:27.531274 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":113,"threads_started":1}
W20260924 04:36:27.532948 9532 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34689 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34689: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 04:36:27.533705 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.524476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.0.98.62:34689)
0924 04:36:27.524712 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.524716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.524729 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:27.524762 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.524764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.524800 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.524889 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.525701 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.526258 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.527082 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.527414 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.527488 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.529191 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:27.529198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:27.529221 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:27.529246 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:27.531324 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:27.531940 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:27.531946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:27.531952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:27.532092 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:27.532367 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:27.532370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:27.532372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:27.532807 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:27.532984 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:27.533312 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 04:36:27.533545 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:27.538786 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.533447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:47791)
0924 04:36:27.533704 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:27.533719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:27.533811 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:27.534052 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:27.534054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:27.534203 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:27.534303 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.534309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.535178 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.535181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.535908 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.535917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.536476 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.536480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.536576 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.536948 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:27.536958 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:27.538707 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184}
W20260924 04:36:27.538954 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:27.533516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47791 (local address 127.0.98.62:34689)
0924 04:36:27.533616 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.533620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.533811 (+ 191us) server_negotiation.cc:407] Connection header received
0924 04:36:27.533895 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.533900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.533940 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.534021 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.534414 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.535070 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.536041 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.536340 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.536451 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.538814 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47791: BlockingRecv error: recv got EOF from 127.0.98.3:47791 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:27 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:27.543751 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:27.539316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:56859)
0924 04:36:27.539425 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:27.539441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:27.539513 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:27.539750 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:27.539753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:27.539893 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:27.539952 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.539957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.540658 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.540661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.541189 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:27.541197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.541608 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.541611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:27.541685 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.542121 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:27.542138 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:27.543669 (+ 1531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 04:36:27.543885 9269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:27.539388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56859 (local address 127.0.98.62:34689)
0924 04:36:27.539569 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:27.539575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:27.539590 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:27.539626 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:27.539627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:27.539655 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:27.539737 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:27.540058 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.540520 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.541295 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:27.541498 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:27.541602 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:27.543780 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56859: BlockingRecv error: recv got EOF from 127.0.98.3:56859 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 04:36:27.544045 9532 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:36:28 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:28 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:28.179790 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:28.174368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:42671)
0924 04:36:28.174651 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:28.174667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:28.174763 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:28.175002 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:28.175005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:28.175190 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:28.175301 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:28.175306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.176057 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.176060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:28.176618 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:28.176626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.177056 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.177059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:28.177155 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:28.177674 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:28.177686 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:28.179661 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
W20260924 04:36:28.179922 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:28.174433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42671 (local address 127.0.98.62:34689)
0924 04:36:28.174655 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:28.174660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:28.174721 (+ 61us) server_negotiation.cc:407] Connection header received
0924 04:36:28.174848 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:28.174851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:28.174893 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:28.174967 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:28.175405 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.175949 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.176732 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.176953 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.177043 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:28.179791 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42671: BlockingRecv error: recv got EOF from 127.0.98.1:42671 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":59,"threads_started":1}
Sep 24 04:36:28 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:28 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:28.364485 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:28.358895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:46225)
0924 04:36:28.359150 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:28.359164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:28.359258 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:28.359608 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:28.359610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:28.359868 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:28.360040 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:28.360068 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.361054 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.361056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:28.361631 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:28.361638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.362025 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.362027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:28.362095 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:28.362552 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:28.362562 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:28.364369 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
W20260924 04:36:28.364722 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:28.358982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46225 (local address 127.0.98.62:34689)
0924 04:36:28.359125 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:28.359129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:28.359271 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:36:28.359314 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:28.359316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:28.359489 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:28.359579 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:28.360189 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.360955 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.361729 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.361934 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.362007 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:28.364490 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46225: BlockingRecv error: recv got EOF from 127.0.98.2:46225 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:36:28 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:28 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:28.549786 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:28.544603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:38713)
0924 04:36:28.544884 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:28.544897 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:28.544994 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:28.545244 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:28.545246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:28.545416 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:28.545517 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:28.545522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.546225 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.546228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:28.546806 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:28.546814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.547229 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.547231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:28.547297 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:28.547782 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:28.547793 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:28.549653 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":118,"threads_started":1}
W20260924 04:36:28.549970 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:28.544691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38713 (local address 127.0.98.62:34689)
0924 04:36:28.544826 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:28.544829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:28.544995 (+ 166us) server_negotiation.cc:407] Connection header received
0924 04:36:28.545065 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:28.545068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:28.545116 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:28.545207 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:28.545615 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.546126 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.546910 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:28.547135 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:28.547212 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:28.549800 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38713: BlockingRecv error: recv got EOF from 127.0.98.3:38713 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 04:36:29 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:29 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:29.186277 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:29.180522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:46537)
0924 04:36:29.180791 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:29.180807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:29.180916 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:29.181185 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:29.181188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:29.181365 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:29.181456 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:29.181461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.182346 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.182350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:29.183219 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:29.183229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.183680 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.183683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:29.183748 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:29.184263 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:29.184275 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:29.186146 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
W20260924 04:36:29.186451 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:29.180611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46537 (local address 127.0.98.62:34689)
0924 04:36:29.180903 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:29.180907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:29.180920 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:29.180986 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:29.180988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:29.181029 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:29.181117 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:29.181575 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.182212 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.183333 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.183558 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.183637 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:29.186282 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46537: BlockingRecv error: recv got EOF from 127.0.98.1:46537 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
Sep 24 04:36:29 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:29 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:29.370647 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:29.365291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:51011)
0924 04:36:29.365533 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:29.365549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:29.365688 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:29.365948 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:29.365951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:29.366168 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:29.366260 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:29.366265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.367039 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.367045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:29.367636 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:29.367644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.368095 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.368099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:29.368186 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:29.368704 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:29.368716 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:29.370469 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
W20260924 04:36:29.370710 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:29.365363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51011 (local address 127.0.98.62:34689)
0924 04:36:29.365495 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:29.365498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:29.365641 (+ 143us) server_negotiation.cc:407] Connection header received
0924 04:36:29.365785 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:29.365788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:29.365840 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:29.365926 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:29.366376 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.366927 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.367754 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.367977 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.368083 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:29.370604 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51011: BlockingRecv error: recv got EOF from 127.0.98.2:51011 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:36:29 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:29 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:29.555472 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:29.550471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:43023)
0924 04:36:29.550766 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:29.550779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:29.550876 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:29.551087 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:29.551089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:29.551267 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:29.551360 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:29.551365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.552073 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.552075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:29.552616 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:29.552622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.553056 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.553058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:29.553116 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:29.553656 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:29.553672 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:29.555342 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
W20260924 04:36:29.555600 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:29.550632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43023 (local address 127.0.98.62:34689)
0924 04:36:29.550782 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:29.550786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:29.550830 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:36:29.550941 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:29.550943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:29.550985 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:29.551053 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:29.551462 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.551983 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.552741 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:29.552963 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:29.553042 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:29.555469 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43023: BlockingRecv error: recv got EOF from 127.0.98.3:43023 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 04:36:30 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:30 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:30.192999 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:30.186997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:47339)
0924 04:36:30.187340 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:30.187355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:30.187448 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:30.187798 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:30.187801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:30.188057 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:30.188231 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:30.188237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.189271 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.189278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:30.189921 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:30.189929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.190442 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.190446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:30.190537 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:30.191026 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:30.191037 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:30.192870 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
W20260924 04:36:30.193209 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:30.187099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47339 (local address 127.0.98.62:34689)
0924 04:36:30.187418 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:30.187421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:30.187435 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:36:30.187615 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:30.187618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:30.187674 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:30.187780 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:30.188370 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.189121 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.190049 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.190320 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.190440 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:30.193000 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47339: BlockingRecv error: recv got EOF from 127.0.98.1:47339 (error 108)
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
Sep 24 04:36:30 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:30 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:30.388054 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:30.371553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:43709)
0924 04:36:30.371852 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:30.371873 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:30.372019 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:30.372294 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:30.372297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:30.372509 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:30.372612 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:30.372618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.373527 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.373532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:30.374104 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:30.374112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.374623 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.374626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:30.374713 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:30.385749 (+ 11036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:30.385763 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:30.387865 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":89,"threads_started":1}
W20260924 04:36:30.388115 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:30.371614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43709 (local address 127.0.98.62:34689)
0924 04:36:30.371819 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:30.371824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:30.371955 (+ 131us) server_negotiation.cc:407] Connection header received
0924 04:36:30.372123 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:30.372127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:30.372176 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:30.372275 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:30.372730 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.373414 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.374226 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.374464 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.374569 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:30.388005 (+ 13436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43709: BlockingRecv error: recv got EOF from 127.0.98.2:43709 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 24 04:36:30 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:30 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:30.562412 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:30.556323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:35799)
0924 04:36:30.556611 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:30.556625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:30.556759 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:30.557012 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:30.557015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:30.557249 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:30.557340 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:30.557346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.558090 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.558095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:30.558675 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:30.558682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.559130 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.559133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:30.559229 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:30.560340 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:30.560351 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:30.562283 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":87,"threads_started":1}
W20260924 04:36:30.562480 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:30.556398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35799 (local address 127.0.98.62:34689)
0924 04:36:30.556541 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:30.556546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:30.556723 (+ 177us) server_negotiation.cc:407] Connection header received
0924 04:36:30.556844 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:30.556848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:30.556903 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:30.556990 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:30.557458 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.557978 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.558793 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:30.559018 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:30.559126 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:30.562391 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35799: BlockingRecv error: recv got EOF from 127.0.98.3:35799 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 04:36:31 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:31 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:31.200485 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:31.193710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:33659)
0924 04:36:31.194033 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:31.194047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:31.194154 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:31.194382 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:31.194384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:31.194566 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:31.194748 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:31.194755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.195472 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.195474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:31.196220 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:31.196232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.196702 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.196706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:31.196789 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:31.198230 (+ 1441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:31.198242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:31.200316 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":106,"threads_started":1}
W20260924 04:36:31.200578 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:31.193794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33659 (local address 127.0.98.62:34689)
0924 04:36:31.194037 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:31.194041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:31.194123 (+ 82us) server_negotiation.cc:407] Connection header received
0924 04:36:31.194229 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:31.194231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:31.194274 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:31.194348 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:31.194848 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.195368 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.196347 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.196565 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.196697 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:31.200441 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33659: BlockingRecv error: recv got EOF from 127.0.98.1:33659 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":58,"threads_started":1}
Sep 24 04:36:31 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:31 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:31.395380 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:31.388991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:53629)
0924 04:36:31.389273 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:31.389296 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:31.389391 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:31.389669 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:31.389672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:31.389850 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:31.389961 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:31.389967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.390756 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.390758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:31.391315 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:31.391322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.391775 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.391777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:31.391836 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:31.393332 (+ 1496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:31.393348 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:31.395259 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":116,"threads_started":1}
W20260924 04:36:31.395506 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:31.389074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53629 (local address 127.0.98.62:34689)
0924 04:36:31.389248 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:31.389252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:31.389366 (+ 114us) server_negotiation.cc:407] Connection header received
0924 04:36:31.389472 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:31.389474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:31.389535 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:31.389640 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:31.390066 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.390653 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.391418 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.391676 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.391759 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:31.395392 (+ 3633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53629: BlockingRecv error: recv got EOF from 127.0.98.2:53629 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 04:36:31 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:31 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:31.568755 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:31.563251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:40519)
0924 04:36:31.563543 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:31.563557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:31.563660 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:31.563871 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:31.563873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:31.564044 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:31.564131 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:31.564136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.564820 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.564822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:31.565391 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:31.565399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.565788 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.565790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:31.565846 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:31.566894 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:31.566905 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:31.568646 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
W20260924 04:36:31.568928 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:31.563366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40519 (local address 127.0.98.62:34689)
0924 04:36:31.563490 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:31.563493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:31.563617 (+ 124us) server_negotiation.cc:407] Connection header received
0924 04:36:31.563727 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:31.563729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:31.563771 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:31.563834 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:31.564226 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.564733 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.565480 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:31.565686 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:31.565768 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:31.568774 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40519: BlockingRecv error: recv got EOF from 127.0.98.3:40519 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 04:36:32 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:32 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:32.216970 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:32.201498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:48789)
0924 04:36:32.201802 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:32.201815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:32.201896 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:32.202173 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:32.202176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:32.202353 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:32.202445 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:32.202450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.203203 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.203206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:32.203793 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:32.203801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.204236 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.204238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:32.204300 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:32.215018 (+ 10718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:32.215031 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:32.216847 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":70,"threads_started":1}
W20260924 04:36:32.217132 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:32.201548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48789 (local address 127.0.98.62:34689)
0924 04:36:32.201841 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:32.201845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:32.201875 (+ 30us) server_negotiation.cc:407] Connection header received
0924 04:36:32.201987 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:32.201989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:32.202032 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:32.202104 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:32.202572 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.203089 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.203913 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.204137 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.204217 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:32.216987 (+ 12770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48789: BlockingRecv error: recv got EOF from 127.0.98.1:48789 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
Sep 24 04:36:32 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:32 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:32.401701 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:32.396224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:55571)
0924 04:36:32.396516 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:32.396531 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:32.396651 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:32.396877 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:32.396879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:32.397060 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:32.397211 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:32.397219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.397930 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.397932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:32.398489 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:32.398496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.398957 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.398960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:32.399021 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:32.399511 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:32.399522 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:32.401479 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
W20260924 04:36:32.401919 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:32.396363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55571 (local address 127.0.98.62:34689)
0924 04:36:32.396520 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:32.396525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:32.396587 (+ 62us) server_negotiation.cc:407] Connection header received
0924 04:36:32.396729 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:32.396731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:32.396775 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:32.396867 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:32.397312 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.397829 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.398613 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.398834 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.398923 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:32.401625 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55571: BlockingRecv error: recv got EOF from 127.0.98.2:55571 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 04:36:32 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:32 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:32.575208 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:32.569715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:49757)
0924 04:36:32.569998 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:32.570011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:32.570105 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:32.570337 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:32.570339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:32.570512 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:32.570603 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:32.570609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.571376 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.571378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:32.571965 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:32.571972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.572396 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.572398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:32.572459 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:32.572939 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:32.572950 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:32.575077 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
W20260924 04:36:32.575336 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:32.569753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49757 (local address 127.0.98.62:34689)
0924 04:36:32.569898 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:32.569902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:32.570084 (+ 182us) server_negotiation.cc:407] Connection header received
0924 04:36:32.570183 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:32.570186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:32.570238 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:32.570303 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:32.570738 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.571272 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.572084 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:32.572300 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:32.572388 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:32.575200 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49757: BlockingRecv error: recv got EOF from 127.0.98.3:49757 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 04:36:33 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:33 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:33.223222 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:33.217898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:60441)
0924 04:36:33.218166 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:33.218235 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:33.218350 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:33.218556 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:33.218558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:33.218842 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:33.218954 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:33.218960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.219695 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.219698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:33.220248 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:33.220255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.220643 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.220645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:33.220716 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:33.221263 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:33.221274 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:33.223104 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
W20260924 04:36:33.223416 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:33.217985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60441 (local address 127.0.98.62:34689)
0924 04:36:33.218221 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:33.218228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:33.218297 (+ 69us) server_negotiation.cc:407] Connection header received
0924 04:36:33.218418 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:33.218420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:33.218464 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:33.218545 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:33.219068 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.219606 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.220344 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.220552 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.220625 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:33.223244 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60441: BlockingRecv error: recv got EOF from 127.0.98.1:60441 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":111,"threads_started":1}
Sep 24 04:36:33 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:33 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:33.408665 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:33.402742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:47987)
0924 04:36:33.403142 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:33.403155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:33.403282 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:33.403554 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:33.403557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:33.403795 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:33.403889 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:33.403895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.404663 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.404667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:33.405333 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:33.405341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.405801 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.405805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:33.405891 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:33.406367 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:33.406381 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:33.408475 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
W20260924 04:36:33.408705 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:33.402794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47987 (local address 127.0.98.62:34689)
0924 04:36:33.402935 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:33.402940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:33.403235 (+ 295us) server_negotiation.cc:407] Connection header received
0924 04:36:33.403387 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:33.403392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:33.403441 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:33.403532 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:33.404010 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.404543 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.405445 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.405686 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.405798 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:33.408592 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47987: BlockingRecv error: recv got EOF from 127.0.98.2:47987 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:36:33 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:33 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:33.581025 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:33.575952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:42651)
0924 04:36:33.576242 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:33.576255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:33.576350 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:33.576558 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:33.576560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:33.576732 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:33.576825 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:33.576830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.577544 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.577547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:33.578106 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:33.578114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.578565 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.578567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:33.578629 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:33.579108 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:33.579120 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:33.580911 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":126,"threads_started":1}
W20260924 04:36:33.581175 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:33.576007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42651 (local address 127.0.98.62:34689)
0924 04:36:33.576149 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:33.576154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:33.576325 (+ 171us) server_negotiation.cc:407] Connection header received
0924 04:36:33.576415 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:33.576417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:33.576461 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:33.576524 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:33.576922 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.577452 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.578241 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:33.578458 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:33.578538 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:33.581030 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42651: BlockingRecv error: recv got EOF from 127.0.98.3:42651 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 04:36:34 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:34 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:34.229851 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:34.223995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:60049)
0924 04:36:34.224337 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:34.224352 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:34.224454 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:34.224748 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:34.224750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:34.224923 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:34.225017 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:34.225022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.226022 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.226026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:34.226843 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:34.226852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.227276 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.227279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:34.227339 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:34.227822 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:34.227833 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:34.229731 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":86,"threads_started":1}
W20260924 04:36:34.230036 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:34.224118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60049 (local address 127.0.98.62:34689)
0924 04:36:34.224364 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:34.224367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:34.224407 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:36:34.224573 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:34.224575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:34.224620 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:34.224698 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:34.225168 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.225831 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.226957 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.227181 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.227264 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:34.229866 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60049: BlockingRecv error: recv got EOF from 127.0.98.1:60049 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":62,"threads_started":1}
Sep 24 04:36:34 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:34 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:34.414831 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:34.409531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:55501)
0924 04:36:34.409847 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:34.409863 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:34.409997 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:34.410327 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:34.410330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:34.410551 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:34.410654 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:34.410660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.411364 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.411366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:34.411906 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:34.411913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.412313 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.412315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:34.412372 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:34.412830 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:34.412841 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:34.414719 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":72,"threads_started":1}
W20260924 04:36:34.414965 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:34.409614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55501 (local address 127.0.98.62:34689)
0924 04:36:34.409770 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:34.409775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:34.409956 (+ 181us) server_negotiation.cc:407] Connection header received
0924 04:36:34.410136 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:34.410140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:34.410194 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:34.410281 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:34.410763 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.411272 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.412004 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.412216 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.412293 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:34.414848 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55501: BlockingRecv error: recv got EOF from 127.0.98.2:55501 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 04:36:34 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:34 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:34.589336 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:34.581895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:44015)
0924 04:36:34.582233 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:34.582250 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:34.582379 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:34.582734 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:34.582738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:34.582968 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:34.583092 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:34.583101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.584148 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.584160 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:34.585361 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:34.585375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.585985 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.585988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:34.586059 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:34.586637 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:34.586651 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:34.589101 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":79,"threads_started":1}
W20260924 04:36:34.589423 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:34.582001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44015 (local address 127.0.98.62:34689)
0924 04:36:34.582133 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:34.582138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:34.582318 (+ 180us) server_negotiation.cc:407] Connection header received
0924 04:36:34.582486 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:34.582495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:34.582582 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:34.582690 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:34.583225 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.584050 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.585469 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:34.585858 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:34.585960 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:34.589300 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44015: BlockingRecv error: recv got EOF from 127.0.98.3:44015 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 04:36:35 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:35 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:35.235913 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:35.230545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:33835)
0924 04:36:35.230841 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:35.230856 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:35.230942 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:35.231153 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:35.231155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:35.231342 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:35.231449 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:35.231454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.232241 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.232243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:35.232838 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:35.232846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.233294 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.233296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:35.233359 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:35.233858 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:35.233868 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:35.235769 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":68,"threads_started":1}
W20260924 04:36:35.236076 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:35.230690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33835 (local address 127.0.98.62:34689)
0924 04:36:35.230900 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:35.230904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:35.230916 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:36:35.231007 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:35.231009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:35.231049 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:35.231129 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:35.231573 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.232132 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.232938 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.233192 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.233267 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:35.235901 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33835: BlockingRecv error: recv got EOF from 127.0.98.1:33835 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
Sep 24 04:36:35 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:35 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:35.421833 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:35.415671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:41695)
0924 04:36:35.415931 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:35.415956 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:35.416049 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:35.416272 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:35.416274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:35.416450 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:35.416567 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:35.416572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.417594 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.417597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:35.418413 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:35.418425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.418986 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.418989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:35.419125 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:35.419621 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:35.419634 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:35.421719 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":65,"threads_started":1}
W20260924 04:36:35.422008 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:35.415794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41695 (local address 127.0.98.62:34689)
0924 04:36:35.416044 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:35.416047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:35.416060 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:35.416123 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:35.416126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:35.416167 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:35.416235 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:35.416673 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.417457 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.418552 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.418841 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.418981 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:35.421882 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41695: BlockingRecv error: recv got EOF from 127.0.98.2:41695 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
Sep 24 04:36:35 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:35 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:35.595717 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:35.590288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:42345)
0924 04:36:35.590535 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:35.590549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:35.590648 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:35.590895 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:35.590898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:35.591077 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:35.591178 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:35.591183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.591927 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.591930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:35.592521 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:35.592530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.593014 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.593016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:35.593091 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:35.593656 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:35.593669 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:35.595564 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
W20260924 04:36:35.595793 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:35.590364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42345 (local address 127.0.98.62:34689)
0924 04:36:35.590489 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:35.590499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:35.590613 (+ 114us) server_negotiation.cc:407] Connection header received
0924 04:36:35.590713 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:35.590715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:35.590760 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:35.590825 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:35.591285 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.591806 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.592692 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:35.592911 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:35.592999 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:35.595703 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42345: BlockingRecv error: recv got EOF from 127.0.98.3:42345 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 04:36:36 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:36 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:36.242035 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:36.236780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:53167)
0924 04:36:36.237041 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:36.237054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:36.237185 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:36.237430 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:36.237432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:36.237599 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:36.237693 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:36.237698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.238413 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.238415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:36.238959 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:36.238967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.239399 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.239401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:36.239464 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:36.239942 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:36.239952 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:36.241924 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
W20260924 04:36:36.242247 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:36.236806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53167 (local address 127.0.98.62:34689)
0924 04:36:36.237084 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:36.237087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:36.237100 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:36.237280 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:36.237282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:36.237325 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:36.237401 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:36.237787 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.238324 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.239065 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.239307 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.239381 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:36.242063 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53167: BlockingRecv error: recv got EOF from 127.0.98.1:53167 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
Sep 24 04:36:36 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:36 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:36.428203 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:36.422566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:52551)
0924 04:36:36.422866 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:36.422878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:36.423013 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:36.423319 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:36.423322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:36.423523 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:36.423626 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:36.423632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.424390 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.424395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:36.424979 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:36.424987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.425520 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.425523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:36.425607 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:36.426073 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:36.426084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:36.428071 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
W20260924 04:36:36.428368 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:36.422635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52551 (local address 127.0.98.62:34689)
0924 04:36:36.422803 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:36.422807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:36.422951 (+ 144us) server_negotiation.cc:407] Connection header received
0924 04:36:36.423121 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:36.423124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:36.423201 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:36.423299 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:36.423744 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.424280 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.425102 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.425393 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.425510 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:36.428218 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52551: BlockingRecv error: recv got EOF from 127.0.98.2:52551 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 04:36:36 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:36 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:36.602372 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:36.596521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:51289)
0924 04:36:36.596773 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:36.596807 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:36.596959 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:36.597237 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:36.597239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:36.597415 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:36.597506 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:36.597511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.598211 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.598213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:36.598818 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:36.598828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.599428 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.599431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:36.599505 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:36.600154 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:36.600165 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:36.602261 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
W20260924 04:36:36.602506 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:36.596597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51289 (local address 127.0.98.62:34689)
0924 04:36:36.596771 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:36.596775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:36.596923 (+ 148us) server_negotiation.cc:407] Connection header received
0924 04:36:36.597059 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:36.597061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:36.597115 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:36.597206 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:36.597607 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.598120 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.598951 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:36.599280 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:36.599393 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:36.602387 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51289: BlockingRecv error: recv got EOF from 127.0.98.3:51289 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 24 04:36:37 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:37 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:37.257725 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:37.242872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:47977)
0924 04:36:37.243163 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:37.243177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:37.243305 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:37.243647 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:37.243650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:37.243854 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:37.243949 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:37.243955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.244732 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.244736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:37.245390 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:37.245399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.245853 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.245857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:37.245942 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:37.255618 (+ 9676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:37.255631 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:37.257520 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
W20260924 04:36:37.257773 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:37.242969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47977 (local address 127.0.98.62:34689)
0924 04:36:37.243193 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:37.243199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:37.243253 (+ 54us) server_negotiation.cc:407] Connection header received
0924 04:36:37.243411 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:37.243416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:37.243475 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:37.243583 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:37.244084 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.244621 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.245506 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.245742 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.245851 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:37.257654 (+ 11803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47977: BlockingRecv error: recv got EOF from 127.0.98.1:47977 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
Sep 24 04:36:37 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:37 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:37.434494 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:37.428994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:48509)
0924 04:36:37.429309 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:37.429323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:37.429420 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:37.429661 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:37.429663 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:37.429836 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:37.429928 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:37.429933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.430637 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.430639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:37.431271 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:37.431279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.431719 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.431729 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:37.431796 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:37.432386 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:37.432397 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:37.434328 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
W20260924 04:36:37.434549 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:37.429064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48509 (local address 127.0.98.62:34689)
0924 04:36:37.429275 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:37.429279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:37.429395 (+ 116us) server_negotiation.cc:407] Connection header received
0924 04:36:37.429512 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:37.429514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:37.429556 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:37.429628 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:37.430026 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.430543 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.431377 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.431594 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.431679 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:37.434443 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48509: BlockingRecv error: recv got EOF from 127.0.98.2:48509 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 04:36:37 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:37 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:37.608284 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:37.603109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:32779)
0924 04:36:37.603372 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:37.603386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:37.603472 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:37.603712 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:37.603715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:37.603887 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:37.603986 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:37.603991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.604700 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.604703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:37.605276 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:37.605284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.605679 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.605682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:37.605741 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:37.606274 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:37.606286 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:37.608116 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
W20260924 04:36:37.608325 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:37.603254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32779 (local address 127.0.98.62:34689)
0924 04:36:37.603380 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:37.603383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:37.603435 (+ 52us) server_negotiation.cc:407] Connection header received
0924 04:36:37.603552 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:37.603554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:37.603609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:37.603676 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:37.604081 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.604580 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.605372 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:37.605576 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:37.605661 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:37.608227 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32779: BlockingRecv error: recv got EOF from 127.0.98.3:32779 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:36:38 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:38 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:38.264201 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:38.258519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:50275)
0924 04:36:38.258820 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:38.258837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:38.258946 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:38.259172 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:38.259174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:38.259370 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:38.259465 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:38.259471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.260223 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.260226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:38.260806 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:38.260814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.261218 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.261220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:38.261290 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:38.261891 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:38.261906 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:38.264019 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":66,"threads_started":1}
W20260924 04:36:38.264261 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:38.258699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50275 (local address 127.0.98.62:34689)
0924 04:36:38.258946 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:38.258950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:38.258963 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:36:38.259017 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:38.259019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:38.259060 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:38.259133 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:38.259600 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.260125 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.260905 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.261117 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.261211 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:38.264139 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50275: BlockingRecv error: recv got EOF from 127.0.98.1:50275 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
Sep 24 04:36:38 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:38 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:38.440213 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:38.435229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:50779)
0924 04:36:38.435480 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:38.435493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:38.435588 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:38.435810 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:38.435812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:38.435986 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:38.436079 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:38.436084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.436813 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.436815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:38.437428 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:38.437436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.437831 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.437833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:38.437890 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:38.438381 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:38.438391 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:38.440090 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
W20260924 04:36:38.440389 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:38.435280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50779 (local address 127.0.98.62:34689)
0924 04:36:38.435436 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:38.435439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:38.435583 (+ 144us) server_negotiation.cc:407] Connection header received
0924 04:36:38.435665 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:38.435667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:38.435713 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:38.435774 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:38.436181 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.436721 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.437528 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.437736 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.437813 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:38.440236 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50779: BlockingRecv error: recv got EOF from 127.0.98.2:50779 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 04:36:38 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:38 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:38.614243 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:38.609098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:49365)
0924 04:36:38.609428 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:38.609441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:38.609552 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:38.609798 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:38.609800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:38.609984 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:38.610084 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:38.610090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.610789 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.610792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:38.611352 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:38.611359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.611741 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.611744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:38.611805 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:38.612288 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:38.612300 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:38.614132 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
W20260924 04:36:38.614411 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:38.609183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49365 (local address 127.0.98.62:34689)
0924 04:36:38.609314 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:38.609317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:38.609515 (+ 198us) server_negotiation.cc:407] Connection header received
0924 04:36:38.609637 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:38.609639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:38.609692 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:38.609759 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:38.610180 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.610681 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.611448 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:38.611652 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:38.611734 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:38.614256 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49365: BlockingRecv error: recv got EOF from 127.0.98.3:49365 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 04:36:39 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:39 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:39.270701 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:39.264984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:56583)
0924 04:36:39.265288 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:39.265302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:39.265394 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:39.265631 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:39.265633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:39.265808 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:39.265955 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:39.265962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.267101 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.267105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:39.267681 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:39.267689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.268100 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.268103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:39.268167 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:39.268758 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:39.268769 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:39.270590 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":74,"threads_started":1}
W20260924 04:36:39.270881 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:39.265108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56583 (local address 127.0.98.62:34689)
0924 04:36:39.265369 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:39.265372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:39.265386 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:36:39.265475 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:39.265477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:39.265519 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:39.265594 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:39.266087 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.266974 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.267780 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.267989 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.268067 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:39.270714 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56583: BlockingRecv error: recv got EOF from 127.0.98.1:56583 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":72,"threads_started":1}
Sep 24 04:36:39 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:39 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:39.446169 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:39.440976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:45607)
0924 04:36:39.441335 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:39.441350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:39.441448 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:39.441663 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:39.441665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:39.441840 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:39.441933 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:39.441938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.442635 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.442637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:39.443182 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:39.443189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.443635 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.443637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:39.443698 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:39.444188 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:39.444199 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:39.446053 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":125,"threads_started":1}
W20260924 04:36:39.446308 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:39.441113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45607 (local address 127.0.98.62:34689)
0924 04:36:39.441285 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:39.441290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:39.441404 (+ 114us) server_negotiation.cc:407] Connection header received
0924 04:36:39.441523 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:39.441525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:39.441569 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:39.441626 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:39.442032 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.442540 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.443301 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.443521 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.443619 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:39.446193 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45607: BlockingRecv error: recv got EOF from 127.0.98.2:45607 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 04:36:39 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:39 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:39.620265 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:39.614961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:41823)
0924 04:36:39.615216 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:39.615232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:39.615339 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:39.615560 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:39.615562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:39.615760 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:39.615854 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:39.615859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.616544 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.616546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:39.617105 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:39.617113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.617588 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.617592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:39.617685 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:39.618174 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:39.618184 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:39.620150 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
W20260924 04:36:39.620453 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:39.615084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41823 (local address 127.0.98.62:34689)
0924 04:36:39.615217 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:39.615221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:39.615308 (+ 87us) server_negotiation.cc:407] Connection header received
0924 04:36:39.615412 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:39.615415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:39.615458 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:39.615528 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:39.615950 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.616458 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.617254 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:39.617476 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:39.617582 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:39.620291 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41823: BlockingRecv error: recv got EOF from 127.0.98.3:41823 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 04:36:40 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:40 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:40.277113 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:40.271441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:42673)
0924 04:36:40.271726 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:40.271740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:40.271830 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:40.272068 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:40.272070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:40.272248 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:40.272342 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:40.272347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.273060 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.273062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:40.273700 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:40.273709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.274149 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.274151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:40.274213 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:40.274699 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:40.274710 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:40.276998 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
W20260924 04:36:40.277307 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:40.271573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42673 (local address 127.0.98.62:34689)
0924 04:36:40.271805 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:40.271809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:40.271821 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:36:40.271916 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:40.271918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:40.271959 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:40.272032 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:40.272435 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.272966 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.273808 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.274038 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.274125 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:40.277119 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42673: BlockingRecv error: recv got EOF from 127.0.98.1:42673 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":59,"threads_started":1}
Sep 24 04:36:40 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:40 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:40.452073 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:40.447006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:52899)
0924 04:36:40.447270 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:40.447283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:40.447368 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:40.447623 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:40.447625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:40.447830 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:40.447924 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:40.447929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.448616 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.448618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:40.449209 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:40.449217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.449623 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.449626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:40.449713 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:40.450179 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:40.450189 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:40.451965 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
W20260924 04:36:40.452246 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:40.447143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52899 (local address 127.0.98.62:34689)
0924 04:36:40.447308 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:40.447311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:40.447325 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:36:40.447489 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:40.447491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:40.447540 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:40.447641 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:40.448021 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.448527 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.449307 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.449528 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.449602 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:40.452086 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52899: BlockingRecv error: recv got EOF from 127.0.98.2:52899 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 04:36:40 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:40 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:40.626035 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:40.620984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:44929)
0924 04:36:40.621305 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:40.621319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:40.621414 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:40.621640 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:40.621642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:40.621817 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:40.621908 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:40.621913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.622594 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.622596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:40.623140 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:40.623147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.623525 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.623527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:40.623585 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:40.624077 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:40.624094 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:40.625926 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
W20260924 04:36:40.626135 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:40.621036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44929 (local address 127.0.98.62:34689)
0924 04:36:40.621201 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:40.621206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:40.621387 (+ 181us) server_negotiation.cc:407] Connection header received
0924 04:36:40.621491 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:40.621493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:40.621539 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:40.621596 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:40.622001 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.622506 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.623233 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:40.623438 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:40.623509 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:40.626045 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44929: BlockingRecv error: recv got EOF from 127.0.98.3:44929 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 04:36:41 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:41 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:41.283445 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:41.277904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:53331)
0924 04:36:41.278247 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:41.278260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:41.278359 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:41.278619 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:41.278621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:41.278797 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:41.278892 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:41.278898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.279880 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.279883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:41.280510 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:41.280520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.280940 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.280942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:41.281004 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:41.281511 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:41.281522 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:41.283297 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":99,"threads_started":1}
W20260924 04:36:41.283623 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:41.277972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53331 (local address 127.0.98.62:34689)
0924 04:36:41.278280 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:41.278283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:41.278310 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:36:41.278435 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:41.278437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:41.278479 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:41.278580 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:41.279011 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.279762 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.280627 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.280843 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.280923 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:41.283431 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53331: BlockingRecv error: recv got EOF from 127.0.98.1:53331 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":90,"threads_started":1}
Sep 24 04:36:41 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:41 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:41.458642 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:41.452816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:55159)
0924 04:36:41.453118 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:41.453155 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:41.453295 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:41.453598 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:41.453601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:41.453804 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:41.453894 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:41.453899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.454663 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.454668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:41.455242 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:41.455250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.455924 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.455928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:41.456021 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:41.456566 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:41.456578 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:41.458455 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":92,"threads_started":1}
W20260924 04:36:41.458736 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:41.452871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55159 (local address 127.0.98.62:34689)
0924 04:36:41.453086 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:41.453091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:41.453247 (+ 156us) server_negotiation.cc:407] Connection header received
0924 04:36:41.453395 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:41.453398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:41.453452 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:41.453540 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:41.454030 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.454553 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.455364 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.455782 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.455922 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:41.458617 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55159: BlockingRecv error: recv got EOF from 127.0.98.2:55159 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 24 04:36:41 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:41 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:41.632447 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:41.626973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:49983)
0924 04:36:41.627250 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:41.627272 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:41.627409 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:41.627694 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:41.627697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:41.627899 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:41.627988 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:41.627993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.628742 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.628748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:41.629349 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:41.629357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.629837 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.629840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:41.629923 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:41.630409 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:41.630420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:41.632260 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
W20260924 04:36:41.632485 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:41.627032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49983 (local address 127.0.98.62:34689)
0924 04:36:41.627218 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:41.627225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:41.627349 (+ 124us) server_negotiation.cc:407] Connection header received
0924 04:36:41.627509 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:41.627513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:41.627562 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:41.627674 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:41.628103 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.628638 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.629468 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:41.629715 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:41.629826 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:41.632388 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49983: BlockingRecv error: recv got EOF from 127.0.98.3:49983 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 04:36:42 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:42 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:42.298218 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:42.284270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:50021)
0924 04:36:42.284529 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:42.284544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:42.284620 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:42.284941 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:42.284945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:42.285198 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:42.285345 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:42.285356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.286324 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.286326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:42.287031 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:42.287041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.287534 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.287537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:42.287597 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:42.295962 (+ 8365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:42.295980 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:42.298079 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20260924 04:36:42.298334 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:42.284324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50021 (local address 127.0.98.62:34689)
0924 04:36:42.284562 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:42.284565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:42.284593 (+ 28us) server_negotiation.cc:407] Connection header received
0924 04:36:42.284739 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:42.284742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:42.284801 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:42.284897 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:42.285473 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.286224 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.287128 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.287410 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.287533 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:42.298210 (+ 10677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50021: BlockingRecv error: recv got EOF from 127.0.98.1:50021 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":61,"threads_started":1}
Sep 24 04:36:42 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:36:42 dist-test-slave-jlnt krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:36:42.464753 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:42.459585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:34721)
0924 04:36:42.459845 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:42.459861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:42.459963 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:42.460225 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:42.460227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:42.460415 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:42.460511 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:42.460517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.461243 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.461245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:42.461814 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:42.461822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.462266 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.462269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:42.462332 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:42.462822 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:42.462833 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:42.464613 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34689: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
W20260924 04:36:42.464912 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:42.459661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34721 (local address 127.0.98.62:34689)
0924 04:36:42.459799 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:42.459803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:42.459938 (+ 135us) server_negotiation.cc:407] Connection header received
0924 04:36:42.460040 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:42.460042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:42.460084 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:42.460152 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:42.460620 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.461153 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.461915 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.462124 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.462201 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:42.464763 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34721: BlockingRecv error: recv got EOF from 127.0.98.2:34721 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:36:42.534432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 9119
I20260924 04:36:42.542122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 9271
I20260924 04:36:42.547128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 9403
I20260924 04:36:42.552233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 9049
2026-09-24T04:36:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17933 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T04:36:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:42Z Disabled control of system clock
I20260924 04:36:42.581638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34893
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:36:42.689656 9599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:42.689883 9599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:42.689916 9599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:42.693742 9599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:42.693784 9599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:42.693799 9599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:42.693821 9599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:42.693837 9599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:42.698378 9599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:42.699244 9599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:42.700467 9599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:42.709080 9607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:42.709199 9605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:42.709164 9599 server_base.cc:1034] running on GCE node
W20260924 04:36:42.709093 9604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:42.709990 9599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:42.711005 9599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:42.712181 9599 hybrid_clock.cc:648] HybridClock initialized: now 1790224602712167 us; error 24 us; skew 500 ppm
I20260924 04:36:42.714062 9599 webserver.cc:460] Webserver started at http://127.0.98.62:45489/ using document root <none> and password file <none>
I20260924 04:36:42.714618 9599 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:42.714679 9599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:42.714890 9599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:42.716820 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "d0ca4cf8c6734a0ca9983d463a2292df"
format_stamp: "Formatted at 2026-09-24 04:36:42 on dist-test-slave-jlnt"
server_key: "85901a2f8a1063680941b33938e236f3"
server_key_iv: "bde137197351ac4569a7326eabc6b355"
server_key_version: "encryptionkey@0"
I20260924 04:36:42.717376 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0ca4cf8c6734a0ca9983d463a2292df"
format_stamp: "Formatted at 2026-09-24 04:36:42 on dist-test-slave-jlnt"
server_key: "85901a2f8a1063680941b33938e236f3"
server_key_iv: "bde137197351ac4569a7326eabc6b355"
server_key_version: "encryptionkey@0"
I20260924 04:36:42.720752 9599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260924 04:36:42.723096 9613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:42.724309 9599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:42.724452 9599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "d0ca4cf8c6734a0ca9983d463a2292df"
format_stamp: "Formatted at 2026-09-24 04:36:42 on dist-test-slave-jlnt"
server_key: "85901a2f8a1063680941b33938e236f3"
server_key_iv: "bde137197351ac4569a7326eabc6b355"
server_key_version: "encryptionkey@0"
I20260924 04:36:42.724589 9599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:42.755656 9599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:42.756371 9599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:42.756577 9599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:42.763351 9599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34893
I20260924 04:36:42.763361 9665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34893 every 8 connection(s)
I20260924 04:36:42.764359 9599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:42.766716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 9599
I20260924 04:36:42.766821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:42.767129 392 external_mini_cluster.cc:1428] Setting key afba3005a03a4942236b991312c81cd9
I20260924 04:36:42.767735 9666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:42.773756 9666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df: Bootstrap starting.
I20260924 04:36:42.775885 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:42.768390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.0.98.62:34893)
0924 04:36:42.768969 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:42.768983 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:42.769017 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:36:42.769723 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:42.769739 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:42.770109 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:42.770429 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:42.770845 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.771769 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.772647 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:42.773098 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:42.773353 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:42.774223 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:42.774227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:42.774238 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:42.774605 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:42.774656 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:42.774675 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:42.774749 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:42.774928 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 04:36:42.775195 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260924 04:36:42.776115 9666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:42.776849 9666 log.cc:824] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:42.778606 9666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df: No bootstrap required, opened a new log
I20260924 04:36:42.781807 9666 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca4cf8c6734a0ca9983d463a2292df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34893 } }
I20260924 04:36:42.781996 9666 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:42.782035 9666 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0ca4cf8c6734a0ca9983d463a2292df, State: Initialized, Role: FOLLOWER
I20260924 04:36:42.782352 9666 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca4cf8c6734a0ca9983d463a2292df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34893 } }
I20260924 04:36:42.782450 9666 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:42.782492 9666 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:42.782563 9666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:42.783268 9666 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca4cf8c6734a0ca9983d463a2292df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34893 } }
I20260924 04:36:42.783623 9666 leader_election.cc:302] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0ca4cf8c6734a0ca9983d463a2292df; no voters:
I20260924 04:36:42.783957 9666 leader_election.cc:288] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:42.784013 9671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:42.784220 9671 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [term 1 LEADER]: Becoming Leader. State: Replica: d0ca4cf8c6734a0ca9983d463a2292df, State: Running, Role: LEADER
I20260924 04:36:42.784525 9671 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca4cf8c6734a0ca9983d463a2292df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34893 } }
I20260924 04:36:42.785172 9666 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:42.786151 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0ca4cf8c6734a0ca9983d463a2292df. Latest consensus state: current_term: 1 leader_uuid: "d0ca4cf8c6734a0ca9983d463a2292df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca4cf8c6734a0ca9983d463a2292df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34893 } } }
I20260924 04:36:42.786278 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:42.786127 9673 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0ca4cf8c6734a0ca9983d463a2292df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ca4cf8c6734a0ca9983d463a2292df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34893 } } }
I20260924 04:36:42.786556 9673 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:42.786643 9680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:42.789175 9680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:42.794459 9680 catalog_manager.cc:1347] Generated new cluster ID: 857474f214bc40f4baf8c609ef9cab4e
I20260924 04:36:42.794528 9680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:42.799201 9680 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:42.799971 9680 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:42.803802 9680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0ca4cf8c6734a0ca9983d463a2292df: Generated new TSK 0
I20260924 04:36:42.804450 9680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:36:42.807060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34893
--builtin_ntp_servers=127.0.98.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 04:36:42.915555 9690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:42.915822 9690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:42.915897 9690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:42.919939 9690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:42.920024 9690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:42.920112 9690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:42.924932 9690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:42.925995 9690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:42.927403 9690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:42.936896 9698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:42.936970 9690 server_base.cc:1034] running on GCE node
W20260924 04:36:42.936954 9695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:42.937125 9696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:42.937497 9690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:42.938100 9690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:42.939286 9690 hybrid_clock.cc:648] HybridClock initialized: now 1790224602939272 us; error 29 us; skew 500 ppm
I20260924 04:36:42.941075 9690 webserver.cc:460] Webserver started at http://127.0.98.1:45409/ using document root <none> and password file <none>
I20260924 04:36:42.941671 9690 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:42.941766 9690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:42.941990 9690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:42.943939 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "52b2c090ea4b4349b4a59166131cab6b"
format_stamp: "Formatted at 2026-09-24 04:36:42 on dist-test-slave-jlnt"
server_key: "6572df283874618b0a1d5d572b8138b9"
server_key_iv: "52c1808cc9996f1f9bb70e912c4ef24b"
server_key_version: "encryptionkey@0"
I20260924 04:36:42.944496 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "52b2c090ea4b4349b4a59166131cab6b"
format_stamp: "Formatted at 2026-09-24 04:36:42 on dist-test-slave-jlnt"
server_key: "6572df283874618b0a1d5d572b8138b9"
server_key_iv: "52c1808cc9996f1f9bb70e912c4ef24b"
server_key_version: "encryptionkey@0"
I20260924 04:36:42.948030 9690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:36:42.950441 9704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:42.951625 9690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:42.951753 9690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "52b2c090ea4b4349b4a59166131cab6b"
format_stamp: "Formatted at 2026-09-24 04:36:42 on dist-test-slave-jlnt"
server_key: "6572df283874618b0a1d5d572b8138b9"
server_key_iv: "52c1808cc9996f1f9bb70e912c4ef24b"
server_key_version: "encryptionkey@0"
I20260924 04:36:42.951860 9690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:42.989866 9690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:42.990607 9690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:42.990813 9690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:42.991425 9690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:42.992372 9690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:42.992444 9690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:42.992515 9690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:42.992558 9690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.001194 9690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40987
I20260924 04:36:43.001210 9817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40987 every 8 connection(s)
I20260924 04:36:43.002127 9690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:43.003245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 9690
I20260924 04:36:43.003372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:43.003743 392 external_mini_cluster.cc:1428] Setting key 4f58f502125e4ba12037777d01ab1293
I20260924 04:36:43.005932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34893
--builtin_ntp_servers=127.0.98.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:36:43.012411 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.004210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56473 (local address 127.0.98.62:34893)
0924 04:36:43.004342 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:43.004347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:43.005761 (+ 1414us) server_negotiation.cc:407] Connection header received
0924 04:36:43.007405 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:43.007408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:43.007458 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:43.007531 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:43.008637 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.009244 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.010431 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.010669 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.010784 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:43.011902 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:43.011905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:43.011908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:43.011921 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:43.011949 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:43.011957 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:43.012038 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:43.012196 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 04:36:43.012260 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:36:43.013299 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.004737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34893 (local address 127.0.98.1:56473)
0924 04:36:43.005615 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:43.005657 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:43.007151 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:43.007731 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:43.007742 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:43.007859 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:43.008496 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:43.008510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.009374 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.009377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:43.010298 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:43.010307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.010821 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.010826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:43.011002 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:43.011638 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:43.011640 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:43.011680 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:43.011743 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:43.012040 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:43.012052 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:43.012059 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:36:43.012196 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 04:36:43.012474 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":219,"threads_started":1}
I20260924 04:36:43.014528 9818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34893
I20260924 04:36:43.014801 9818 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:43.015492 9818 heartbeater.cc:511] Master 127.0.98.62:34893 requested a full tablet report, sending...
I20260924 04:36:43.017132 9630 ts_manager.cc:194] Registered new tserver with Master: 52b2c090ea4b4349b4a59166131cab6b (127.0.98.1:40987)
I20260924 04:36:43.018334 9630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56473
W20260924 04:36:43.116171 9821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:43.116421 9821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:43.116456 9821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:43.120792 9821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:43.120853 9821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:43.120908 9821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:43.125698 9821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:43.126739 9821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:43.128135 9821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:43.137458 9826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:43.137547 9827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:43.137459 9829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:43.137530 9821 server_base.cc:1034] running on GCE node
I20260924 04:36:43.138064 9821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:43.138885 9821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:43.140095 9821 hybrid_clock.cc:648] HybridClock initialized: now 1790224603140076 us; error 30 us; skew 500 ppm
I20260924 04:36:43.141981 9821 webserver.cc:460] Webserver started at http://127.0.98.2:43595/ using document root <none> and password file <none>
I20260924 04:36:43.142544 9821 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:43.142642 9821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:43.142868 9821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:43.144842 9821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "ffe5a4495fae4542be5fb0dbabc6fb05"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "0d2796c04081e3d98a566994a6489570"
server_key_iv: "656d2d86a9139c6a676b415006605e3e"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.145419 9821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ffe5a4495fae4542be5fb0dbabc6fb05"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "0d2796c04081e3d98a566994a6489570"
server_key_iv: "656d2d86a9139c6a676b415006605e3e"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.148926 9821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:36:43.151412 9835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.152649 9821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:43.152787 9821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "ffe5a4495fae4542be5fb0dbabc6fb05"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "0d2796c04081e3d98a566994a6489570"
server_key_iv: "656d2d86a9139c6a676b415006605e3e"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.152899 9821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:43.179165 9821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:43.179853 9821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:43.180049 9821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:43.180627 9821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:43.181602 9821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:43.181680 9821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.181746 9821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:43.181792 9821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.190524 9821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41093
I20260924 04:36:43.190549 9948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41093 every 8 connection(s)
I20260924 04:36:43.191383 9821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:43.192512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 9821
I20260924 04:36:43.192618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:43.192929 392 external_mini_cluster.cc:1428] Setting key 270dbcea6aabc9f3a07c43be8c62bf5a
I20260924 04:36:43.194942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34893
--builtin_ntp_servers=127.0.98.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:36:43.200717 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.193304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49813 (local address 127.0.98.62:34893)
0924 04:36:43.193439 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:43.193444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:43.194643 (+ 1199us) server_negotiation.cc:407] Connection header received
0924 04:36:43.196057 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:43.196061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:43.196121 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:43.196198 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:43.197056 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.197664 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.198777 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.199007 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.199125 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:43.200224 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:43.200227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:43.200230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:43.200249 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:43.200266 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:43.200274 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:43.200350 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:43.200503 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 04:36:43.200561 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:36:43.201583 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.193812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34893 (local address 127.0.98.2:49813)
0924 04:36:43.194491 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:43.194540 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:43.195860 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:43.196360 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:43.196368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:43.196449 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:43.196921 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:43.196932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.197786 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.197791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:43.198654 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:43.198663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.199161 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.199165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:43.199327 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:43.199979 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:43.199981 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:43.200019 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:43.200078 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:43.200363 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:43.200375 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:43.200385 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 04:36:43.200502 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:36:43.200775 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":131,"threads_started":1}
I20260924 04:36:43.202729 9949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34893
I20260924 04:36:43.202988 9949 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:43.203635 9949 heartbeater.cc:511] Master 127.0.98.62:34893 requested a full tablet report, sending...
I20260924 04:36:43.204849 9630 ts_manager.cc:194] Registered new tserver with Master: ffe5a4495fae4542be5fb0dbabc6fb05 (127.0.98.2:41093)
I20260924 04:36:43.205615 9630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49813
W20260924 04:36:43.305063 9952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:43.305349 9952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:43.305421 9952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:43.309342 9952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:43.309387 9952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:43.309437 9952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:43.313973 9952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:43.314846 9952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:43.316090 9952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:43.325214 9960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:43.325317 9952 server_base.cc:1034] running on GCE node
W20260924 04:36:43.325165 9958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:43.325197 9957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:43.325822 9952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:43.326357 9952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:43.327680 9952 hybrid_clock.cc:648] HybridClock initialized: now 1790224603327646 us; error 46 us; skew 500 ppm
I20260924 04:36:43.329596 9952 webserver.cc:460] Webserver started at http://127.0.98.3:43229/ using document root <none> and password file <none>
I20260924 04:36:43.330132 9952 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:43.330190 9952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:43.330425 9952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:43.332355 9952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "9677d484fabb4209a1c24b4c5e29b8f9"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "a1ceff204a3f69c4968511d0a4489540"
server_key_iv: "9b8f6d7e053886d4c748cb41c4197fd2"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.332891 9952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9677d484fabb4209a1c24b4c5e29b8f9"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "a1ceff204a3f69c4968511d0a4489540"
server_key_iv: "9b8f6d7e053886d4c748cb41c4197fd2"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.336434 9952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:36:43.338850 9966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.340025 9952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:43.340179 9952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "9677d484fabb4209a1c24b4c5e29b8f9"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "a1ceff204a3f69c4968511d0a4489540"
server_key_iv: "9b8f6d7e053886d4c748cb41c4197fd2"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.340288 9952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:43.353849 9952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:43.354439 9952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:43.354619 9952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:43.355232 9952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:43.356231 9952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:43.356293 9952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.356354 9952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:43.356384 9952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.364527 9952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43869
I20260924 04:36:43.364559 10079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43869 every 8 connection(s)
I20260924 04:36:43.365619 9952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:43.370556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 9952
I20260924 04:36:43.370689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:43.370985 392 external_mini_cluster.cc:1428] Setting key 8be4d50a601543eebcaf3bfa8e62bf6a
I20260924 04:36:43.374814 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.367342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58585 (local address 127.0.98.62:34893)
0924 04:36:43.367489 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:43.367494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:43.368551 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 04:36:43.369947 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:43.369950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:43.369995 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:43.370068 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:43.370970 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.371611 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.372966 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.373233 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.373314 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:43.374419 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:43.374422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:43.374424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:43.374436 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:43.374454 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:43.374461 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:43.374529 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:43.374662 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:36:43.374710 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:36:43.375595 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.367792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34893 (local address 127.0.98.3:58585)
0924 04:36:43.368407 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:43.368445 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:43.369758 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:43.370222 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:43.370228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:43.370306 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:43.370837 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:43.370851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.371738 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.371745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:43.372787 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:43.372795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:43.373393 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:43.373397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:43.373538 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:43.374182 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:43.374184 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:43.374221 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:43.374279 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:43.374539 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:43.374544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:43.374549 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:36:43.374639 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:36:43.374845 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260924 04:36:43.376585 10080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34893
I20260924 04:36:43.376781 10080 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:43.377282 10080 heartbeater.cc:511] Master 127.0.98.62:34893 requested a full tablet report, sending...
I20260924 04:36:43.378185 9630 ts_manager.cc:194] Registered new tserver with Master: 9677d484fabb4209a1c24b4c5e29b8f9 (127.0.98.3:43869)
I20260924 04:36:43.378712 9630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58585
I20260924 04:36:43.384562 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:36:43.387384 9669 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:43.386589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.0.98.62:34893)
0924 04:36:43.386725 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:43.386729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:43.386879 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:36:43.386955 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:43.386957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:43.386993 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:43.387096 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:43.387298 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55866: BlockingRecv error: recv got EOF from 127.0.0.1:55866 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:36:43.387596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 9690
I20260924 04:36:43.393699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 9821
I20260924 04:36:43.398833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 9952
I20260924 04:36:43.403903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 9599
2026-09-24T04:36:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (850 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T04:36:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:43Z Disabled control of system clock
I20260924 04:36:43.431978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42875
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:36:43.538288 10091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:43.538549 10091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:43.538616 10091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:43.542584 10091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:43.542637 10091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:43.542652 10091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:43.542670 10091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:43.542685 10091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:43.547369 10091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:43.548288 10091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:43.549619 10091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:43.557634 10096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:43.557777 10091 server_base.cc:1034] running on GCE node
W20260924 04:36:43.557822 10099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:43.557634 10097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:43.558465 10091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:43.559544 10091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:43.560735 10091 hybrid_clock.cc:648] HybridClock initialized: now 1790224603560703 us; error 47 us; skew 500 ppm
I20260924 04:36:43.562587 10091 webserver.cc:460] Webserver started at http://127.0.98.62:35127/ using document root <none> and password file <none>
I20260924 04:36:43.563133 10091 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:43.563220 10091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:43.563446 10091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:43.565460 10091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "302a4d9486af42ae86b3cce9bfb810cd"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "47a1b7f0b528474e4cf4cd6de951c755"
server_key_iv: "819ad433ba4ce2d183068917c41071fb"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.566056 10091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "302a4d9486af42ae86b3cce9bfb810cd"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "47a1b7f0b528474e4cf4cd6de951c755"
server_key_iv: "819ad433ba4ce2d183068917c41071fb"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.569523 10091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:36:43.571736 10105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.572818 10091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:43.572958 10091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "302a4d9486af42ae86b3cce9bfb810cd"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "47a1b7f0b528474e4cf4cd6de951c755"
server_key_iv: "819ad433ba4ce2d183068917c41071fb"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.573060 10091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:43.577502 10091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:43.578047 10091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:43.578222 10091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:43.584452 10091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42875
I20260924 04:36:43.584478 10157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42875 every 8 connection(s)
I20260924 04:36:43.585340 10091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:43.586899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 10091
I20260924 04:36:43.587049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:43.587333 392 external_mini_cluster.cc:1428] Setting key 6d8b9dda9f026d6466dee747c37bed7f
I20260924 04:36:43.588536 10158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:43.593358 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.588570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.0.98.62:42875)
0924 04:36:43.589285 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:43.589305 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:43.589346 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:36:43.590190 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:43.590211 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:43.590625 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:43.590981 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:43.591908 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:43.591914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:43.591929 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:43.591942 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:43.592013 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:43.592039 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:43.592192 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:43.592259 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 04:36:43.592486 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260924 04:36:43.595160 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd: Bootstrap starting.
I20260924 04:36:43.597251 10158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:43.597940 10158 log.cc:824] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:43.599745 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd: No bootstrap required, opened a new log
I20260924 04:36:43.603067 10158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302a4d9486af42ae86b3cce9bfb810cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42875 } }
I20260924 04:36:43.603295 10158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:43.603385 10158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302a4d9486af42ae86b3cce9bfb810cd, State: Initialized, Role: FOLLOWER
I20260924 04:36:43.603793 10158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302a4d9486af42ae86b3cce9bfb810cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42875 } }
I20260924 04:36:43.603960 10158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:43.604024 10158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:43.604120 10158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:43.605026 10158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302a4d9486af42ae86b3cce9bfb810cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42875 } }
I20260924 04:36:43.605372 10158 leader_election.cc:302] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 302a4d9486af42ae86b3cce9bfb810cd; no voters:
I20260924 04:36:43.605665 10158 leader_election.cc:288] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:43.605788 10163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:43.606053 10163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [term 1 LEADER]: Becoming Leader. State: Replica: 302a4d9486af42ae86b3cce9bfb810cd, State: Running, Role: LEADER
I20260924 04:36:43.606369 10163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302a4d9486af42ae86b3cce9bfb810cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42875 } }
I20260924 04:36:43.606822 10158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:43.607898 10165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 302a4d9486af42ae86b3cce9bfb810cd. Latest consensus state: current_term: 1 leader_uuid: "302a4d9486af42ae86b3cce9bfb810cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302a4d9486af42ae86b3cce9bfb810cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42875 } } }
I20260924 04:36:43.608019 10165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:43.608343 10164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "302a4d9486af42ae86b3cce9bfb810cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302a4d9486af42ae86b3cce9bfb810cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42875 } } }
I20260924 04:36:43.608476 10164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:43.608420 10172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:43.611512 10172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:43.616613 10172 catalog_manager.cc:1347] Generated new cluster ID: 0ec03809ceb04f78b895a40e5d5fe079
I20260924 04:36:43.616677 10172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:43.632395 10172 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:43.632683 10172 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:43.640525 10172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 302a4d9486af42ae86b3cce9bfb810cd: Generated new TSK 0
I20260924 04:36:43.641125 10172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:36:43.645952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42875
--builtin_ntp_servers=127.0.98.20:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:36:43.754627 10182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:43.754853 10182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:43.754891 10182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:43.758895 10182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:43.758939 10182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:43.759023 10182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:43.763674 10182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:43.764622 10182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:43.766063 10182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:36:43.775310 10182 server_base.cc:1034] running on GCE node
W20260924 04:36:43.775262 10187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:43.775281 10188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:43.775266 10190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:43.775928 10182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:43.776532 10182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:43.777720 10182 hybrid_clock.cc:648] HybridClock initialized: now 1790224603777680 us; error 50 us; skew 500 ppm
I20260924 04:36:43.779482 10182 webserver.cc:460] Webserver started at http://127.0.98.1:33645/ using document root <none> and password file <none>
I20260924 04:36:43.780046 10182 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:43.780135 10182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:43.780355 10182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:43.782351 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "b7f1909af5904982ada85756cc66963f"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "33323dae07fed921db2ce55ffd041c01"
server_key_iv: "bd345c45df75c162c53984517e12aab2"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.782905 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b7f1909af5904982ada85756cc66963f"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "33323dae07fed921db2ce55ffd041c01"
server_key_iv: "bd345c45df75c162c53984517e12aab2"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.786365 10182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:36:43.788586 10196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.789794 10182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:43.789933 10182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "b7f1909af5904982ada85756cc66963f"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "33323dae07fed921db2ce55ffd041c01"
server_key_iv: "bd345c45df75c162c53984517e12aab2"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.790040 10182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:43.794356 10182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:43.794907 10182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:43.795089 10182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:43.795596 10182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:43.796439 10182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:43.796509 10182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.796586 10182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:43.796630 10182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.804899 10182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42633
I20260924 04:36:43.804921 10309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42633 every 8 connection(s)
I20260924 04:36:43.805788 10182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:43.811676 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.807566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53099 (local address 127.0.98.62:42875)
0924 04:36:43.807706 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:43.807709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:43.808762 (+ 1053us) server_negotiation.cc:407] Connection header received
0924 04:36:43.810129 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:43.810132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:43.810174 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:43.810250 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:43.811247 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:43.811250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:43.811252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:43.811263 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:43.811280 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:43.811288 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:43.811360 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:43.811518 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 04:36:43.811557 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:36:43.811816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 10182
I20260924 04:36:43.811937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:43.812244 392 external_mini_cluster.cc:1428] Setting key 191817842dd4f30bf106cf75d72e362b
I20260924 04:36:43.812691 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.807946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42875 (local address 127.0.98.1:53099)
0924 04:36:43.808621 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:43.808659 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:43.809934 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:43.810396 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:43.810403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:43.810484 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:43.810989 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:43.810991 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:43.811028 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:43.811106 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:43.811362 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:43.811368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:43.811371 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:36:43.811495 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 04:36:43.811748 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":153,"threads_started":1}
I20260924 04:36:43.814005 10310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42875
I20260924 04:36:43.814251 10310 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:43.814318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42875
--builtin_ntp_servers=127.0.98.20:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:36:43.814666 10310 heartbeater.cc:511] Master 127.0.98.62:42875 requested a full tablet report, sending...
I20260924 04:36:43.816067 10121 ts_manager.cc:194] Registered new tserver with Master: b7f1909af5904982ada85756cc66963f (127.0.98.1:42633)
W20260924 04:36:43.919785 10313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:43.920028 10313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:43.920065 10313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:43.923918 10313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:43.923960 10313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:43.924012 10313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:43.928596 10313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:43.929562 10313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:43.930827 10313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:43.939867 10321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:43.939910 10318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:43.939915 10319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:43.940004 10313 server_base.cc:1034] running on GCE node
I20260924 04:36:43.940497 10313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:43.941109 10313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:43.942337 10313 hybrid_clock.cc:648] HybridClock initialized: now 1790224603942307 us; error 42 us; skew 500 ppm
I20260924 04:36:43.944017 10313 webserver.cc:460] Webserver started at http://127.0.98.2:34825/ using document root <none> and password file <none>
I20260924 04:36:43.944573 10313 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:43.944664 10313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:43.944885 10313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:43.946875 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "9e545f6af8c74600a3d0a0037b3fc31b"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "824a813791e208b5d261faff96223dd6"
server_key_iv: "08b294c191a9b09fdbc85922b994c47c"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.947445 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9e545f6af8c74600a3d0a0037b3fc31b"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "824a813791e208b5d261faff96223dd6"
server_key_iv: "08b294c191a9b09fdbc85922b994c47c"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.951022 10313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:36:43.953428 10327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.954605 10313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:36:43.954752 10313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "9e545f6af8c74600a3d0a0037b3fc31b"
format_stamp: "Formatted at 2026-09-24 04:36:43 on dist-test-slave-jlnt"
server_key: "824a813791e208b5d261faff96223dd6"
server_key_iv: "08b294c191a9b09fdbc85922b994c47c"
server_key_version: "encryptionkey@0"
I20260924 04:36:43.954864 10313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:43.959348 10313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:43.959959 10313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:43.960136 10313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:43.960650 10313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:43.961506 10313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:43.961577 10313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.961654 10313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:43.961697 10313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:43.970252 10313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37093
I20260924 04:36:43.970273 10440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37093 every 8 connection(s)
I20260924 04:36:43.971069 10313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:43.976887 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.972920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32939 (local address 127.0.98.62:42875)
0924 04:36:43.973082 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:43.973087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:43.974100 (+ 1013us) server_negotiation.cc:407] Connection header received
0924 04:36:43.975292 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:43.975295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:43.975330 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:43.975406 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:43.976467 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:43.976470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:43.976471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:43.976482 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:43.976501 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:43.976509 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:43.976585 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:43.976750 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:36:43.976788 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:36:43.977732 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:43.973360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42875 (local address 127.0.98.2:32939)
0924 04:36:43.973963 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:43.973999 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:43.975099 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:43.975558 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:43.975564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:43.975648 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:43.976231 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:43.976232 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:43.976268 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:43.976332 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:43.976611 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:43.976619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:43.976626 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:36:43.976765 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 04:36:43.976992 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260924 04:36:43.978785 10441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42875
I20260924 04:36:43.978962 10441 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:43.979310 10441 heartbeater.cc:511] Master 127.0.98.62:42875 requested a full tablet report, sending...
I20260924 04:36:43.980173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 10313
I20260924 04:36:43.980300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:43.980301 10121 ts_manager.cc:194] Registered new tserver with Master: 9e545f6af8c74600a3d0a0037b3fc31b (127.0.98.2:37093)
I20260924 04:36:43.980584 392 external_mini_cluster.cc:1428] Setting key a860ab1dbbc8229ff84bd0d5bc0817fc
I20260924 04:36:43.982498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42875
--builtin_ntp_servers=127.0.98.20:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:36:44.090696 10444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:44.090925 10444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:44.090962 10444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:44.095091 10444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:44.095136 10444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:44.095194 10444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:44.099750 10444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:44.100625 10444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:44.101923 10444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:44.111060 10449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:44.111060 10450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:44.111238 10444 server_base.cc:1034] running on GCE node
W20260924 04:36:44.111239 10452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:44.111807 10444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:44.112339 10444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:44.113483 10444 hybrid_clock.cc:648] HybridClock initialized: now 1790224604113452 us; error 43 us; skew 500 ppm
I20260924 04:36:44.115135 10444 webserver.cc:460] Webserver started at http://127.0.98.3:44085/ using document root <none> and password file <none>
I20260924 04:36:44.115671 10444 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:44.115732 10444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:44.115989 10444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:44.117959 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "e56d7f0ffb784a5aa978c35a22f8f419"
format_stamp: "Formatted at 2026-09-24 04:36:44 on dist-test-slave-jlnt"
server_key: "cf9a1087ecb91e37f5b5c85e711c1422"
server_key_iv: "ac4105af3370f9d486b3395904061ed7"
server_key_version: "encryptionkey@0"
I20260924 04:36:44.118494 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e56d7f0ffb784a5aa978c35a22f8f419"
format_stamp: "Formatted at 2026-09-24 04:36:44 on dist-test-slave-jlnt"
server_key: "cf9a1087ecb91e37f5b5c85e711c1422"
server_key_iv: "ac4105af3370f9d486b3395904061ed7"
server_key_version: "encryptionkey@0"
I20260924 04:36:44.121948 10444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:36:44.124199 10458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:44.125439 10444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:36:44.125527 10444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "e56d7f0ffb784a5aa978c35a22f8f419"
format_stamp: "Formatted at 2026-09-24 04:36:44 on dist-test-slave-jlnt"
server_key: "cf9a1087ecb91e37f5b5c85e711c1422"
server_key_iv: "ac4105af3370f9d486b3395904061ed7"
server_key_version: "encryptionkey@0"
I20260924 04:36:44.125643 10444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:44.129908 10444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:44.130450 10444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:44.130625 10444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:44.131129 10444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:44.131944 10444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:44.131989 10444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:44.132051 10444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:44.132090 10444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:44.140875 10444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33533
I20260924 04:36:44.140933 10571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33533 every 8 connection(s)
I20260924 04:36:44.141762 10444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:44.147416 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:44.143467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57793 (local address 127.0.98.62:42875)
0924 04:36:44.143638 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:44.143641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:44.144724 (+ 1083us) server_negotiation.cc:407] Connection header received
0924 04:36:44.145987 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:44.145989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:44.146023 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:44.146095 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:44.147049 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:44.147052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:44.147055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:36:44.147065 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:44.147083 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:36:44.147091 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:44.147164 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:44.147282 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:36:44.147316 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:36:44.147771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 10444
I20260924 04:36:44.147882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:44.148149 392 external_mini_cluster.cc:1428] Setting key e5b03aadc693341ddf9fe2745b363e08
I20260924 04:36:44.148335 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:44.143853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42875 (local address 127.0.98.3:57793)
0924 04:36:44.144541 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:44.144578 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:44.145811 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:44.146218 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:44.146224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:44.146307 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:44.146817 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:36:44.146818 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:44.146855 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:36:44.146916 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:44.147169 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:44.147175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:44.147178 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:36:44.147300 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 04:36:44.147504 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260924 04:36:44.149370 10572 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42875
I20260924 04:36:44.149605 10572 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:44.150048 10572 heartbeater.cc:511] Master 127.0.98.62:42875 requested a full tablet report, sending...
I20260924 04:36:44.150971 10122 ts_manager.cc:194] Registered new tserver with Master: e56d7f0ffb784a5aa978c35a22f8f419 (127.0.98.3:33533)
I20260924 04:36:44.161810 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:36:44.164351 10161 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:36:44.163594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.0.98.62:42875)
0924 04:36:44.163711 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:44.163714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:44.163843 (+ 129us) server_negotiation.cc:407] Connection header received
0924 04:36:44.163943 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:44.163945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:44.163974 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:44.164033 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:44.164219 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59108: BlockingRecv error: recv got EOF from 127.0.0.1:59108 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:36:44.164714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 10182
I20260924 04:36:44.170375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 10313
I20260924 04:36:44.175473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 10444
I20260924 04:36:44.180405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 10091
2026-09-24T04:36:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (776 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:44 dist-test-slave-jlnt krb5kdc[10582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:44 dist-test-slave-jlnt krb5kdc[10582](info): set up 2 sockets
Sep 24 04:36:44 dist-test-slave-jlnt krb5kdc[10582](info): commencing operation
krb5kdc: starting...
W20260924 04:36:45.242115 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:45 dist-test-slave-jlnt krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:36:45.360186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38607
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:45.466810 10595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:45.467115 10595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:45.467190 10595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:45.471261 10595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:36:45.471343 10595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:45.471381 10595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:45.471442 10595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:36:45.471481 10595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:36:45.476706 10595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:45.477715 10595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:45.478686 10595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:36:45.486863 10595 server_base.cc:1034] running on GCE node
W20260924 04:36:45.486824 10600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:45.486846 10601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:45.486829 10603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:45.487720 10595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:45.488776 10595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:45.489936 10595 hybrid_clock.cc:648] HybridClock initialized: now 1790224605489903 us; error 44 us; skew 500 ppm
Sep 24 04:36:45 dist-test-slave-jlnt krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:45.492530 10595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:36:45.493459 10595 webserver.cc:460] Webserver started at http://127.0.98.62:40033/ using document root <none> and password file <none>
I20260924 04:36:45.493990 10595 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:45.494050 10595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:45.494261 10595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:45.496220 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "744587289b2e40bcae4c7da9cdf93b16"
format_stamp: "Formatted at 2026-09-24 04:36:45 on dist-test-slave-jlnt"
server_key: "ae0d2418495451e08c3ca4d21bb4bfb0"
server_key_iv: "46fdcd645e8f9146e986d7480c40edff"
server_key_version: "encryptionkey@0"
I20260924 04:36:45.496762 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "744587289b2e40bcae4c7da9cdf93b16"
format_stamp: "Formatted at 2026-09-24 04:36:45 on dist-test-slave-jlnt"
server_key: "ae0d2418495451e08c3ca4d21bb4bfb0"
server_key_iv: "46fdcd645e8f9146e986d7480c40edff"
server_key_version: "encryptionkey@0"
I20260924 04:36:45.500244 10595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:36:45.502480 10610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:45.503679 10595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:36:45.503823 10595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "744587289b2e40bcae4c7da9cdf93b16"
format_stamp: "Formatted at 2026-09-24 04:36:45 on dist-test-slave-jlnt"
server_key: "ae0d2418495451e08c3ca4d21bb4bfb0"
server_key_iv: "46fdcd645e8f9146e986d7480c40edff"
server_key_version: "encryptionkey@0"
I20260924 04:36:45.503921 10595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:45.515617 10595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:45.518473 10595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:45.518671 10595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:45.525449 10595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38607
I20260924 04:36:45.525736 10662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38607 every 8 connection(s)
I20260924 04:36:45.526391 10595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:36:45.529017 10663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:45.534742 10663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16: Bootstrap starting.
I20260924 04:36:45.535632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 10595
I20260924 04:36:45.535727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:36:45.536005 392 external_mini_cluster.cc:1428] Setting key 84270e32637e7bcaa6168ef8319e959a
I20260924 04:36:45.537432 10663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:45.538137 10663 log.cc:824] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:45.540177 10663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16: No bootstrap required, opened a new log
I20260924 04:36:45.544526 10663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744587289b2e40bcae4c7da9cdf93b16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } }
Sep 24 04:36:45 dist-test-slave-jlnt krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:45.544763 10663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:45.544855 10663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744587289b2e40bcae4c7da9cdf93b16, State: Initialized, Role: FOLLOWER
I20260924 04:36:45.545326 10663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744587289b2e40bcae4c7da9cdf93b16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } }
I20260924 04:36:45.545475 10663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:36:45.545557 10663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:36:45.545670 10663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:45.546486 10663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744587289b2e40bcae4c7da9cdf93b16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } }
I20260924 04:36:45.546814 10663 leader_election.cc:302] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 744587289b2e40bcae4c7da9cdf93b16; no voters:
I20260924 04:36:45.547098 10663 leader_election.cc:288] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:36:45.547204 10668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:45.547410 10668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [term 1 LEADER]: Becoming Leader. State: Replica: 744587289b2e40bcae4c7da9cdf93b16, State: Running, Role: LEADER
I20260924 04:36:45.547724 10668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744587289b2e40bcae4c7da9cdf93b16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } }
I20260924 04:36:45.548436 10663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:36:45.549872 10669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "744587289b2e40bcae4c7da9cdf93b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744587289b2e40bcae4c7da9cdf93b16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } } }
I20260924 04:36:45.549996 10669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:45.550369 10678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:36:45.550859 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:45.537131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.0.98.62:38607)
0924 04:36:45.537589 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:45.537614 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:45.537665 (+ 51us) server_negotiation.cc:407] Connection header received
0924 04:36:45.538293 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:45.538315 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:45.538665 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:45.538934 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:45.539761 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.540784 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.541981 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.542436 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.542559 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:45.545437 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:45.545442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:45.545473 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:45.545507 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:45.547532 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:45.548083 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:45.548089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:45.548095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:45.548234 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:45.548607 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:45.548610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:45.548612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:45.549061 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:45.549293 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:45.549557 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 04:36:45.549872 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":158,"threads_started":1}
I20260924 04:36:45.551481 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 744587289b2e40bcae4c7da9cdf93b16. Latest consensus state: current_term: 1 leader_uuid: "744587289b2e40bcae4c7da9cdf93b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744587289b2e40bcae4c7da9cdf93b16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } } }
I20260924 04:36:45.551575 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16 [sys.catalog]: This master's current role is: LEADER
I20260924 04:36:45.553781 10678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:36:45.559046 10678 catalog_manager.cc:1347] Generated new cluster ID: d4daa7be2ddf4068b2434502fc4e8d29
I20260924 04:36:45.559115 10678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:36:45.571620 10678 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:36:45.572459 10678 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:36:45.579635 10678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16: Generated new TSK 0
I20260924 04:36:45.580255 10678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:36:45.606225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:45.715085 10689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:45.715318 10689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:45.715360 10689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:45.719372 10689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:45.719426 10689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:45.719515 10689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:36:45.724571 10689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:45.725637 10689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:45.726622 10689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:45.735950 10697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:45.735970 10695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:45.736007 10694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:45.736116 10689 server_base.cc:1034] running on GCE node
I20260924 04:36:45.736476 10689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:45.737090 10689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:45.738307 10689 hybrid_clock.cc:648] HybridClock initialized: now 1790224605738294 us; error 26 us; skew 500 ppm
Sep 24 04:36:45 dist-test-slave-jlnt krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:45.740981 10689 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:36:45.741880 10689 webserver.cc:460] Webserver started at http://127.0.98.1:34969/ using document root <none> and password file <none>
I20260924 04:36:45.742384 10689 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:45.742439 10689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:45.742597 10689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:45.744513 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad373f514c1744ebba4a63cdb27fedf0"
format_stamp: "Formatted at 2026-09-24 04:36:45 on dist-test-slave-jlnt"
server_key: "e3b4b00d38c61fa2d4d8b9a08ee558dd"
server_key_iv: "9c30a95a4a299e7dbd567cefb773f0b2"
server_key_version: "encryptionkey@0"
I20260924 04:36:45.745002 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad373f514c1744ebba4a63cdb27fedf0"
format_stamp: "Formatted at 2026-09-24 04:36:45 on dist-test-slave-jlnt"
server_key: "e3b4b00d38c61fa2d4d8b9a08ee558dd"
server_key_iv: "9c30a95a4a299e7dbd567cefb773f0b2"
server_key_version: "encryptionkey@0"
I20260924 04:36:45.748662 10689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 04:36:45.751031 10704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:45.752271 10689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:45.752388 10689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "ad373f514c1744ebba4a63cdb27fedf0"
format_stamp: "Formatted at 2026-09-24 04:36:45 on dist-test-slave-jlnt"
server_key: "e3b4b00d38c61fa2d4d8b9a08ee558dd"
server_key_iv: "9c30a95a4a299e7dbd567cefb773f0b2"
server_key_version: "encryptionkey@0"
I20260924 04:36:45.752487 10689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:45.776607 10689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:45.779739 10689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:45.779937 10689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:45.780517 10689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:45.781342 10689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:45.781387 10689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:45.781447 10689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:45.781478 10689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:45.790252 10689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33915
I20260924 04:36:45.790270 10817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33915 every 8 connection(s)
I20260924 04:36:45.791255 10689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:36:45.792157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 10689
I20260924 04:36:45.792248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:36:45.792534 392 external_mini_cluster.cc:1428] Setting key c99e9a2712ec3588fef2938aa4cf72f7
Sep 24 04:36:45 dist-test-slave-jlnt krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:36:45.805892 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:45.793318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48383 (local address 127.0.98.62:38607)
0924 04:36:45.793443 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:45.793448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:45.794271 (+ 823us) server_negotiation.cc:407] Connection header received
0924 04:36:45.795526 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:45.795529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:45.795580 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:45.795665 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:45.796755 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.797661 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.798924 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.799384 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.799519 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:45.802250 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:45.802253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:45.802255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:45.802280 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:45.803739 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:45.804183 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:45.804185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:45.804186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:45.804256 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:45.804616 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:45.804620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:45.804621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:45.804807 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:45.804893 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:45.805533 (+ 640us) server_negotiation.cc:299] Negotiation successful
0924 04:36:45.805662 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:36:45.806576 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:45.793626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38607 (local address 127.0.98.1:48383)
0924 04:36:45.794143 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:45.794183 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:45.795331 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:45.795829 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:45.795835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:45.796175 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:45.796619 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:45.796631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.797795 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.797798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:45.798802 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:45.798812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.799537 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.799540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:45.799699 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:45.800287 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:45.800302 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:45.802067 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:45.803839 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:45.803844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:45.803854 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:45.804087 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:45.804334 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:45.804337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:45.804339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:45.804495 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:45.804902 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:45.804907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:45.805254 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 04:36:45.805457 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 04:36:45.805723 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":139,"threads_started":1}
I20260924 04:36:45.807796 10818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38607
I20260924 04:36:45.808038 10818 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:45.808502 10818 heartbeater.cc:511] Master 127.0.98.62:38607 requested a full tablet report, sending...
I20260924 04:36:45.810046 10627 ts_manager.cc:194] Registered new tserver with Master: ad373f514c1744ebba4a63cdb27fedf0 (127.0.98.1:33915)
I20260924 04:36:45.811143 10627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:36:45.821719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:45.929252 10823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:45.929491 10823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:45.929579 10823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:45.933562 10823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:45.933611 10823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:45.933686 10823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:36:45.938678 10823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:45.939616 10823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:45.940579 10823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:36:45.949841 10828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:45.949858 10829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:45.949894 10823 server_base.cc:1034] running on GCE node
W20260924 04:36:45.949862 10831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:45.950403 10823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:45.951073 10823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:45.952250 10823 hybrid_clock.cc:648] HybridClock initialized: now 1790224605952221 us; error 43 us; skew 500 ppm
Sep 24 04:36:45 dist-test-slave-jlnt krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:45.954973 10823 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:36:45.955746 10823 webserver.cc:460] Webserver started at http://127.0.98.2:36677/ using document root <none> and password file <none>
I20260924 04:36:45.956341 10823 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:45.956426 10823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:45.956642 10823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:45.958640 10823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc"
format_stamp: "Formatted at 2026-09-24 04:36:45 on dist-test-slave-jlnt"
server_key: "354b8ea5446ba1904823c1875cef75a8"
server_key_iv: "181c5168ec8108e71677ec3ce895c03f"
server_key_version: "encryptionkey@0"
I20260924 04:36:45.959184 10823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc"
format_stamp: "Formatted at 2026-09-24 04:36:45 on dist-test-slave-jlnt"
server_key: "354b8ea5446ba1904823c1875cef75a8"
server_key_iv: "181c5168ec8108e71677ec3ce895c03f"
server_key_version: "encryptionkey@0"
I20260924 04:36:45.962714 10823 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:36:45.964927 10838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:45.966114 10823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:36:45.966241 10823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc"
format_stamp: "Formatted at 2026-09-24 04:36:45 on dist-test-slave-jlnt"
server_key: "354b8ea5446ba1904823c1875cef75a8"
server_key_iv: "181c5168ec8108e71677ec3ce895c03f"
server_key_version: "encryptionkey@0"
I20260924 04:36:45.966339 10823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:45.977821 10823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:45.980778 10823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:45.980960 10823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:45.981518 10823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:45.982249 10823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:45.982316 10823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:45.982381 10823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:45.982424 10823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:45.990806 10823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33409
I20260924 04:36:45.990837 10951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33409 every 8 connection(s)
I20260924 04:36:45.991631 10823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:36:45.997512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 10823
I20260924 04:36:45.997610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:36:45.997908 392 external_mini_cluster.cc:1428] Setting key 1f61a48f6e418bba6209ebad76c55f82
Sep 24 04:36:45 dist-test-slave-jlnt krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:46.005590 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:45.993424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46579 (local address 127.0.98.62:38607)
0924 04:36:45.993563 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:45.993566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:45.994396 (+ 830us) server_negotiation.cc:407] Connection header received
0924 04:36:45.995700 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:45.995704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:45.995757 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:45.995869 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:45.996959 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.997584 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.998790 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.999013 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.999138 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.001981 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:46.001984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:46.001986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:46.002049 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:46.003585 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:46.004095 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:46.004099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:46.004101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:46.004184 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:46.004511 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:46.004516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:46.004518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:46.004706 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:46.004811 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:46.005322 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 04:36:46.005436 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:36:46.006413 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:45.993828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38607 (local address 127.0.98.2:46579)
0924 04:36:45.994255 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:45.994294 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:45.995469 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:45.996032 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:45.996038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:45.996373 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:45.996821 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:45.996834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.997738 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.997741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:45.998663 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:45.998671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:45.999161 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:45.999165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:45.999336 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:45.999933 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:45.999949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:46.001808 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:46.003719 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:46.003725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:46.003740 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:46.003990 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:46.004277 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:46.004281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:46.004283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:46.004418 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:46.004808 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:46.004815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:46.005058 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 04:36:46.005281 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 04:36:46.005593 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:36:46.007697 10952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38607
I20260924 04:36:46.007934 10952 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:46.008545 10952 heartbeater.cc:511] Master 127.0.98.62:38607 requested a full tablet report, sending...
I20260924 04:36:46.009955 10627 ts_manager.cc:194] Registered new tserver with Master: ce77a6b2dd474e0c9b7e93471bce3fbc (127.0.98.2:33409)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:36:46.010723 10627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:36:46.026499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:36:46.134965 10957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:36:46.135188 10957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:36:46.135228 10957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:36:46.139168 10957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:36:46.139214 10957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:36:46.139309 10957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:36:46.144362 10957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:36:46.145327 10957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:36:46.146401 10957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:36:46.155727 10957 server_base.cc:1034] running on GCE node
W20260924 04:36:46.155694 10962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:46.155810 10963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:36:46.155701 10965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:36:46.156284 10957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:36:46.156922 10957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:36:46.158114 10957 hybrid_clock.cc:648] HybridClock initialized: now 1790224606158077 us; error 46 us; skew 500 ppm
Sep 24 04:36:46 dist-test-slave-jlnt krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:36:46.160672 10957 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:36:46.161597 10957 webserver.cc:460] Webserver started at http://127.0.98.3:45337/ using document root <none> and password file <none>
I20260924 04:36:46.162175 10957 fs_manager.cc:359] Metadata directory not provided
I20260924 04:36:46.162266 10957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:36:46.162488 10957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:36:46.164446 10957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "23be21ece2984fffa50919c241aec58e"
format_stamp: "Formatted at 2026-09-24 04:36:46 on dist-test-slave-jlnt"
server_key: "3441f3b86abfbf5d7748ac7ae318c967"
server_key_iv: "dd1e9c515a6b5c4fb82eb35fbb8fbd3c"
server_key_version: "encryptionkey@0"
I20260924 04:36:46.164996 10957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "23be21ece2984fffa50919c241aec58e"
format_stamp: "Formatted at 2026-09-24 04:36:46 on dist-test-slave-jlnt"
server_key: "3441f3b86abfbf5d7748ac7ae318c967"
server_key_iv: "dd1e9c515a6b5c4fb82eb35fbb8fbd3c"
server_key_version: "encryptionkey@0"
I20260924 04:36:46.168536 10957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:36:46.171041 10972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:46.172362 10957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:36:46.172583 10957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "23be21ece2984fffa50919c241aec58e"
format_stamp: "Formatted at 2026-09-24 04:36:46 on dist-test-slave-jlnt"
server_key: "3441f3b86abfbf5d7748ac7ae318c967"
server_key_iv: "dd1e9c515a6b5c4fb82eb35fbb8fbd3c"
server_key_version: "encryptionkey@0"
I20260924 04:36:46.172700 10957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:36:46.195289 10957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:36:46.198057 10957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:36:46.198248 10957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:36:46.198792 10957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:36:46.199558 10957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:36:46.199635 10957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:46.199702 10957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:36:46.199739 10957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:36:46.208324 10957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33545
I20260924 04:36:46.208362 11085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33545 every 8 connection(s)
I20260924 04:36:46.209173 10957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:36:46.212316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 10957
I20260924 04:36:46.212409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:36:46.212687 392 external_mini_cluster.cc:1428] Setting key 1e6bd992409595775d628650c932e34d
Sep 24 04:36:46 dist-test-slave-jlnt krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:36:46.222223 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.210848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47621 (local address 127.0.98.62:38607)
0924 04:36:46.211006 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:46.211010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:46.211803 (+ 793us) server_negotiation.cc:407] Connection header received
0924 04:36:46.213194 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:46.213197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:46.213251 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:46.213350 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:46.214940 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.215475 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.216525 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.216741 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.216827 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.219121 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:46.219124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:46.219126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:46.219148 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:46.220407 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:46.220855 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:46.220857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:46.220859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:46.220929 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:46.221359 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:46.221361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:46.221362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:46.221535 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:46.221617 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:46.222013 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 04:36:46.222113 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:36:46.222950 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.211215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38607 (local address 127.0.98.3:47621)
0924 04:36:46.211652 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:46.211693 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:46.212919 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:46.213532 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:46.213539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:46.214035 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:36:46.214677 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.214699 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.215583 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.215586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.216419 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.216427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.216865 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.216868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.216993 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.217534 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:36:46.217549 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:36:46.218968 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:36:46.220502 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:46.220506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:46.220515 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:46.220770 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:46.221006 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:36:46.221008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:36:46.221009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:36:46.221129 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:36:46.221620 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:36:46.221624 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:36:46.221823 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 04:36:46.221966 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 04:36:46.222176 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
I20260924 04:36:46.223986 11086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38607
I20260924 04:36:46.224220 11086 heartbeater.cc:464] Registering TS with master...
I20260924 04:36:46.224674 11086 heartbeater.cc:511] Master 127.0.98.62:38607 requested a full tablet report, sending...
I20260924 04:36:46.225732 10627 ts_manager.cc:194] Registered new tserver with Master: 23be21ece2984fffa50919c241aec58e (127.0.98.3:33545)
I20260924 04:36:46.226338 10627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47621
I20260924 04:36:46.226605 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:36:46.235579 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.228656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.0.98.62:38607)
0924 04:36:46.228781 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:46.228785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:46.228943 (+ 158us) server_negotiation.cc:407] Connection header received
0924 04:36:46.229070 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:46.229074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:46.229112 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:46.229220 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:36:46.229870 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.230361 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.231196 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.231427 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.231528 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.232719 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:36:46.232723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:36:46.232725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:36:46.232748 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:36:46.234007 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:46.234411 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:46.234415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:46.234417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:46.234491 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:36:46.234765 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:36:46.234768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:36:46.234769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:36:46.234963 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:36:46.235098 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:46.235329 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:36:46.235427 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:36:46.238207 10627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:36:46.241241 10627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:36:46.255432 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.249056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33915 (local address 127.0.0.1:53642)
0924 04:36:46.249842 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:46.249890 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:46.250010 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:46.251269 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:46.251272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:46.251284 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:46.251406 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.251415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.252632 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.252635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.253862 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.253873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.254919 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.254923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.255028 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.255065 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 04:36:46.255183 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:36:46.255289 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":48,"thread_start_us":88,"threads_started":1}
I20260924 04:36:46.255456 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.249227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.0.98.1:33915)
0924 04:36:46.250865 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:46.250873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:46.250891 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:36:46.250950 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:46.250953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:46.251152 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:46.251303 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:46.251516 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.252260 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.253987 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.254793 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.254882 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.254988 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:46.255223 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 04:36:46.255317 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1556,"thread_start_us":132,"threads_started":1}
I20260924 04:36:46.256244 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.249288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33545 (local address 127.0.0.1:45098)
0924 04:36:46.250397 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:46.250408 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:46.250519 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:46.251221 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:46.251225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:46.251243 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:46.252328 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.252362 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.253564 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.253569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.254624 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.254654 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.255892 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.255894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.255955 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.255971 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:36:46.256061 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:36:46.256126 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":984,"thread_start_us":108,"threads_started":1}
I20260924 04:36:46.256341 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.249419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.0.98.3:33545)
0924 04:36:46.249986 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:46.249998 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:46.250504 (+ 506us) server_negotiation.cc:407] Connection header received
0924 04:36:46.250682 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:46.250688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:46.250889 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:46.251017 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:46.252449 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.253442 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.254768 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.255546 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.255685 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.255824 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:46.256107 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 04:36:46.256197 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":140,"threads_started":1}
I20260924 04:36:46.257220 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.250059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33409 (local address 127.0.0.1:49264)
0924 04:36:46.250483 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:46.250498 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:46.250580 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:46.251100 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:46.251105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:46.251125 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:46.251277 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.251284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.252305 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.252311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.253866 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.253873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.256805 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.256809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.256899 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.256922 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:36:46.256991 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:36:46.257063 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":69,"thread_start_us":102,"threads_started":1}
I20260924 04:36:46.257694 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.250313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.0.98.2:33409)
0924 04:36:46.250704 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:46.250717 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:46.250738 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:36:46.250801 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:46.250805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:46.250946 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:46.251083 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:46.251374 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.252174 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.255081 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.256280 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.256621 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.256745 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:46.257464 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 04:36:46.257552 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
I20260924 04:36:46.258839 11020 tablet_service.cc:1467] Processing CreateTablet for tablet e077468a0281473e93be3d9c8df5fd88 (DEFAULT_TABLE table=test-table [id=cb3625beb64642a1aac4c91efc7f9f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:46.258836 10752 tablet_service.cc:1467] Processing CreateTablet for tablet e077468a0281473e93be3d9c8df5fd88 (DEFAULT_TABLE table=test-table [id=cb3625beb64642a1aac4c91efc7f9f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:46.259832 10752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e077468a0281473e93be3d9c8df5fd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:46.259843 11020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e077468a0281473e93be3d9c8df5fd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:46.265489 11103 tablet_bootstrap.cc:492] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0: Bootstrap starting.
I20260924 04:36:46.266855 10886 tablet_service.cc:1467] Processing CreateTablet for tablet e077468a0281473e93be3d9c8df5fd88 (DEFAULT_TABLE table=test-table [id=cb3625beb64642a1aac4c91efc7f9f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:36:46.267433 11103 tablet_bootstrap.cc:654] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:46.267722 10886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e077468a0281473e93be3d9c8df5fd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:36:46.268487 11103 log.cc:824] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:46.270496 11103 tablet_bootstrap.cc:492] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0: No bootstrap required, opened a new log
I20260924 04:36:46.270699 11103 ts_tablet_manager.cc:1397] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:36:46.273209 11104 tablet_bootstrap.cc:492] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: Bootstrap starting.
I20260924 04:36:46.273754 11106 tablet_bootstrap.cc:492] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: Bootstrap starting.
I20260924 04:36:46.273743 11103 raft_consensus.cc:348] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } }
I20260924 04:36:46.274024 11103 raft_consensus.cc:374] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:46.274080 11103 raft_consensus.cc:729] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad373f514c1744ebba4a63cdb27fedf0, State: Initialized, Role: FOLLOWER
I20260924 04:36:46.274524 11103 consensus_queue.cc:260] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } }
I20260924 04:36:46.275154 10818 heartbeater.cc:503] Master 127.0.98.62:38607 was elected leader, sending a full tablet report...
I20260924 04:36:46.275341 11104 tablet_bootstrap.cc:654] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:46.275494 11103 ts_tablet_manager.cc:1428] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:36:46.276547 11104 log.cc:824] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:46.277274 11106 tablet_bootstrap.cc:654] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: Neither blocks nor log segments found. Creating new log.
I20260924 04:36:46.277915 11106 log.cc:824] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:36:46.279542 11104 tablet_bootstrap.cc:492] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: No bootstrap required, opened a new log
I20260924 04:36:46.279742 11104 ts_tablet_manager.cc:1397] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260924 04:36:46.279784 11106 tablet_bootstrap.cc:492] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: No bootstrap required, opened a new log
I20260924 04:36:46.279951 11106 ts_tablet_manager.cc:1397] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260924 04:36:46.282528 11104 raft_consensus.cc:348] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } }
I20260924 04:36:46.282562 11106 raft_consensus.cc:348] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } }
I20260924 04:36:46.282755 11104 raft_consensus.cc:374] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:46.282747 11106 raft_consensus.cc:374] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:36:46.282814 11106 raft_consensus.cc:729] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce77a6b2dd474e0c9b7e93471bce3fbc, State: Initialized, Role: FOLLOWER
I20260924 04:36:46.282815 11104 raft_consensus.cc:729] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23be21ece2984fffa50919c241aec58e, State: Initialized, Role: FOLLOWER
I20260924 04:36:46.283210 11106 consensus_queue.cc:260] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } }
I20260924 04:36:46.283211 11104 consensus_queue.cc:260] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } }
I20260924 04:36:46.283804 11086 heartbeater.cc:503] Master 127.0.98.62:38607 was elected leader, sending a full tablet report...
I20260924 04:36:46.283807 10952 heartbeater.cc:503] Master 127.0.98.62:38607 was elected leader, sending a full tablet report...
I20260924 04:36:46.284109 11106 ts_tablet_manager.cc:1428] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260924 04:36:46.284350 11104 ts_tablet_manager.cc:1428] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 04:36:46.292418 10819 tablet.cc:2369] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:36:46.460103 11087 tablet.cc:2369] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:36:46.492730 10953 tablet.cc:2369] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:36:46.686338 11110 raft_consensus.cc:482] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:36:46.686579 11110 raft_consensus.cc:504] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } }
I20260924 04:36:46.687731 11110 leader_election.cc:288] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad373f514c1744ebba4a63cdb27fedf0 (127.0.98.1:33915), 23be21ece2984fffa50919c241aec58e (127.0.98.3:33545)
I20260924 04:36:46.693252 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.688189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51301 (local address 127.0.98.1:33915)
0924 04:36:46.688312 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:46.688317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:46.688525 (+ 208us) server_negotiation.cc:407] Connection header received
0924 04:36:46.688699 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:46.688703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:46.688759 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:46.688847 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:46.689296 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.690110 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.691914 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.692533 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.692662 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.692730 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:46.693021 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 04:36:46.693103 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:36:46.693252 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.688151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57317 (local address 127.0.98.3:33545)
0924 04:36:46.688289 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:46.688294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:46.688684 (+ 390us) server_negotiation.cc:407] Connection header received
0924 04:36:46.688738 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:46.688741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:46.688794 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:46.688868 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:36:46.689296 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.690117 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.691914 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.692592 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.692684 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.692732 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:46.693021 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 04:36:46.693103 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:36:46.693279 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.688003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33545 (local address 127.0.98.2:57317)
0924 04:36:46.688498 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:46.688511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:46.688595 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:46.688886 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:46.688890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:46.688928 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:46.689172 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.689178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.690236 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.690239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.691780 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.691791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.692704 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.692708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.692834 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.692896 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 04:36:46.692971 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:36:46.693073 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":341,"spinlock_wait_cycles":27136,"thread_start_us":62,"threads_started":1}
I20260924 04:36:46.693281 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.688003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33915 (local address 127.0.98.2:51301)
0924 04:36:46.688412 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:46.688427 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:36:46.688555 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:36:46.688886 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:36:46.688890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:36:46.688931 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:36:46.689165 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.689173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.690236 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.690239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.691780 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:36:46.691791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.692704 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.692708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:36:46.692838 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.692890 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 04:36:46.692971 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:36:46.693073 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":247,"spinlock_wait_cycles":9856,"thread_start_us":106,"threads_started":1}
I20260924 04:36:46.694022 11040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e077468a0281473e93be3d9c8df5fd88" candidate_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23be21ece2984fffa50919c241aec58e" is_pre_election: true
I20260924 04:36:46.694182 10772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e077468a0281473e93be3d9c8df5fd88" candidate_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad373f514c1744ebba4a63cdb27fedf0" is_pre_election: true
I20260924 04:36:46.694356 11040 raft_consensus.cc:2454] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce77a6b2dd474e0c9b7e93471bce3fbc in term 0.
I20260924 04:36:46.694409 10772 raft_consensus.cc:2454] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce77a6b2dd474e0c9b7e93471bce3fbc in term 0.
I20260924 04:36:46.694842 10839 leader_election.cc:302] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23be21ece2984fffa50919c241aec58e, ad373f514c1744ebba4a63cdb27fedf0, ce77a6b2dd474e0c9b7e93471bce3fbc; no voters:
I20260924 04:36:46.695168 11110 raft_consensus.cc:2781] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:36:46.695251 11110 raft_consensus.cc:482] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:36:46.695302 11110 raft_consensus.cc:3037] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:46.696296 11110 raft_consensus.cc:504] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } }
I20260924 04:36:46.696676 11110 leader_election.cc:288] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [CANDIDATE]: Term 1 election: Requested vote from peers ad373f514c1744ebba4a63cdb27fedf0 (127.0.98.1:33915), 23be21ece2984fffa50919c241aec58e (127.0.98.3:33545)
I20260924 04:36:46.696977 10772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e077468a0281473e93be3d9c8df5fd88" candidate_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad373f514c1744ebba4a63cdb27fedf0"
I20260924 04:36:46.697062 11040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e077468a0281473e93be3d9c8df5fd88" candidate_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23be21ece2984fffa50919c241aec58e"
I20260924 04:36:46.697109 10772 raft_consensus.cc:3037] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:46.697237 11040 raft_consensus.cc:3037] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:36:46.698127 10772 raft_consensus.cc:2454] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce77a6b2dd474e0c9b7e93471bce3fbc in term 1.
I20260924 04:36:46.698295 11040 raft_consensus.cc:2454] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce77a6b2dd474e0c9b7e93471bce3fbc in term 1.
I20260924 04:36:46.698415 10842 leader_election.cc:302] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad373f514c1744ebba4a63cdb27fedf0, ce77a6b2dd474e0c9b7e93471bce3fbc; no voters:
I20260924 04:36:46.698616 11110 raft_consensus.cc:2781] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:36:46.698809 11110 raft_consensus.cc:686] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 1 LEADER]: Becoming Leader. State: Replica: ce77a6b2dd474e0c9b7e93471bce3fbc, State: Running, Role: LEADER
I20260924 04:36:46.699113 11110 consensus_queue.cc:237] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } }
I20260924 04:36:46.702112 10625 catalog_manager.cc:5495] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc reported cstate change: term changed from 0 to 1, leader changed from <none> to ce77a6b2dd474e0c9b7e93471bce3fbc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce77a6b2dd474e0c9b7e93471bce3fbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33409 } health_report { overall_health: HEALTHY } } }
I20260924 04:36:46.812655 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.809089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.0.98.2:33409)
0924 04:36:46.809378 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:36:46.809382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:36:46.809397 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:36:46.809469 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:36:46.809472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:36:46.809543 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:36:46.809660 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:36:46.809871 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.810385 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.811233 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:36:46.811472 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:36:46.811569 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:36:46.811827 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:36:46.812209 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:36:46.812303 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:36:46.812459 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 04:36:46.812517 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":149,"threads_started":1}
I20260924 04:36:46.818194 10772 raft_consensus.cc:1261] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [term 1 FOLLOWER]: Refusing update from remote peer ce77a6b2dd474e0c9b7e93471bce3fbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:46.818194 11040 raft_consensus.cc:1261] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [term 1 FOLLOWER]: Refusing update from remote peer ce77a6b2dd474e0c9b7e93471bce3fbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:36:46.818876 11114 consensus_queue.cc:1035] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "23be21ece2984fffa50919c241aec58e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:46.819044 11110 consensus_queue.cc:1035] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad373f514c1744ebba4a63cdb27fedf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:36:46.827956 11122 mvcc.cc:205] Tried to move back new op lower bound from 7332759989520367616 to 7332759989042057216. Current Snapshot: MvccSnapshot[applied={T|T < 7332759989520367616}]
I20260924 04:36:46.834265 10625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:46.834393 10625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:36:46.836586 10625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 744587289b2e40bcae4c7da9cdf93b16: Sending DeleteTablet for 3 replicas of tablet e077468a0281473e93be3d9c8df5fd88
I20260924 04:36:46.837322 11020 tablet_service.cc:1514] Processing DeleteTablet for tablet e077468a0281473e93be3d9c8df5fd88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45098
I20260924 04:36:46.837321 10752 tablet_service.cc:1514] Processing DeleteTablet for tablet e077468a0281473e93be3d9c8df5fd88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53642
I20260924 04:36:46.837467 10886 tablet_service.cc:1514] Processing DeleteTablet for tablet e077468a0281473e93be3d9c8df5fd88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:36:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49264
I20260924 04:36:46.837862 11133 tablet_replica.cc:330] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: stopping tablet replica
I20260924 04:36:46.838131 11131 tablet_replica.cc:330] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0: stopping tablet replica
I20260924 04:36:46.838164 11133 raft_consensus.cc:2229] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:36:46.838366 11131 raft_consensus.cc:2229] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:46.838488 11133 raft_consensus.cc:2258] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:46.838639 11131 raft_consensus.cc:2258] T e077468a0281473e93be3d9c8df5fd88 P ad373f514c1744ebba4a63cdb27fedf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:46.838694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 10689
I20260924 04:36:46.839318 11132 tablet_replica.cc:330] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: stopping tablet replica
I20260924 04:36:46.839583 11132 raft_consensus.cc:2229] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:36:46.839841 11132 raft_consensus.cc:2258] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:36:46.841708 11132 ts_tablet_manager.cc:1905] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:46.841859 11133 ts_tablet_manager.cc:1905] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:36:46.844388 11133 ts_tablet_manager.cc:1918] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:46.844412 11132 ts_tablet_manager.cc:1918] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:36:46.844491 11133 log.cc:1196] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/e077468a0281473e93be3d9c8df5fd88
I20260924 04:36:46.844625 11132 log.cc:1196] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/e077468a0281473e93be3d9c8df5fd88
I20260924 04:36:46.844808 11133 ts_tablet_manager.cc:1939] T e077468a0281473e93be3d9c8df5fd88 P ce77a6b2dd474e0c9b7e93471bce3fbc: Deleting consensus metadata
I20260924 04:36:46.845049 11132 ts_tablet_manager.cc:1939] T e077468a0281473e93be3d9c8df5fd88 P 23be21ece2984fffa50919c241aec58e: Deleting consensus metadata
I20260924 04:36:46.845960 10614 catalog_manager.cc:4841] TS ce77a6b2dd474e0c9b7e93471bce3fbc (127.0.98.2:33409): tablet e077468a0281473e93be3d9c8df5fd88 (table test-table [id=cb3625beb64642a1aac4c91efc7f9f07]) successfully deleted
I20260924 04:36:46.846263 10611 catalog_manager.cc:4841] TS 23be21ece2984fffa50919c241aec58e (127.0.98.3:33545): tablet e077468a0281473e93be3d9c8df5fd88 (table test-table [id=cb3625beb64642a1aac4c91efc7f9f07]) successfully deleted
W20260924 04:36:46.846834 10614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33915 (error 108)
I20260924 04:36:46.847055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 10823
I20260924 04:36:46.848245 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:36:46.847865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33915 (local address 127.0.0.1:53654)
0924 04:36:46.848092 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 04:36:46.848159 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
W20260924 04:36:46.848446 10614 catalog_manager.cc:4584] TS ad373f514c1744ebba4a63cdb27fedf0 (127.0.98.1:33915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e077468a0281473e93be3d9c8df5fd88: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33915: connect: Connection refused (error 111)
I20260924 04:36:46.853456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 10957
I20260924 04:36:46.858670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 10595
2026-09-24T04:36:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2679 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:36:46 dist-test-slave-jlnt krb5kdc[11138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:36:46 dist-test-slave-jlnt krb5kdc[11138](info): set up 2 sockets
Sep 24 04:36:46 dist-test-slave-jlnt krb5kdc[11138](info): commencing operation
krb5kdc: starting...
W20260924 04:36:47.902333 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:36:47 dist-test-slave-jlnt krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:36:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:36:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:36:48.068117 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVN1buQ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:36:50.538 UTC [11168] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:36:50.539 UTC [11168] LOG: listening on IPv4 address "127.0.98.20", port 59889
2026-09-24 04:36:50.542 UTC [11168] LOG: listening on Unix socket "/tmp/.s.PGSQL.59889"
2026-09-24 04:36:50.577 UTC [11171] LOG: database system was shut down at 2026-09-24 04:36:49 UTC
2026-09-24 04:36:50.581 UTC [11168] LOG: database system is ready to accept connections
I20260924 04:36:51.521961 392 mini_postgres.cc:96] Postgres bound to 59889
2026-09-24 04:36:51.528 UTC [11179] FATAL: database "slave" does not exist
127.0.98.20:59889 - accepting connections
I20260924 04:36:51.528934 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:36:51.543432 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:36:52.018316 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:36:52.018415 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-admin
2026-09-24 04:36:52,303 [I] DB FLAVOR :POSTGRES
2026-09-24 04:36:52,303 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:36:52,304 [I] Checking connection..
2026-09-24 04:36:52,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:36:52,538 [I] Checking connection passed.
2026-09-24 04:36:52,538 [I] --------- Verifying version history table ---------
2026-09-24 04:36:52,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:36:52,776 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:36:52,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:36:53,031 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:36:53,031 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:36:53,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:36:53.260 UTC [11243] WARNING: there is no transaction in progress
2026-09-24 04:36:53,266 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:36:53,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:36:53,500 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:36:53,500 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:36:53,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:36:53,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:36:53,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jlnt.c.gcp-upstream.internal','N') ;"
2026-09-24 04:36:54,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:36:54,466 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:36:54,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:36:54,704 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:36:54,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:36:54,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:36:55,172 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:36:55,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:36:56.151 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.167 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.183 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.199 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.216 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.233 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.285 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.292 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.305 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.314 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.324 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.337 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.346 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.351 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.358 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56.364 UTC [11351] WARNING: there is no transaction in progress
2026-09-24 04:36:56,986 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:36:56,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:36:57,229 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:36:57,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:36:57,462 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:36:57,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:36:57,711 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:36:57,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:36:57,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:36:58,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jlnt.c.gcp-upstream.internal';"
2026-09-24 04:36:58,407 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:36:58,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:36:58,646 [I] DB_PATCHES have already been applied
I20260924 04:36:58.652009 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVN1buQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:36:58.652066 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:36:58.655233 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40683
Sep 24, 2026 4:36:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:36:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:36:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:36:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:36:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:36:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:36:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:36:59.767649 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.239s user 0.000s sys 0.010s
Sep 24, 2026 4:37:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:37:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 4:37:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:37:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:37:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:37:21.194377 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:37:21 dist-test-slave-jlnt krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224641, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:37:21.284132 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:37:21.294571 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:37:22.172199 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:37:22.172300 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-kms
I20260924 04:37:22.172389 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:59889
2026-09-24 04:37:22,352 [I] DB FLAVOR :POSTGRES
2026-09-24 04:37:22,352 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:37:22,352 [I] Checking connection
2026-09-24 04:37:22,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:37:22,594 [I] connection success
2026-09-24 04:37:22,594 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:37:22,594 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:37:22,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:37:22,821 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:37:22,821 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:37:22,821 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:37:22,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59889/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:37:23,071 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:37:23.543712 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:37:23.727195 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:37:23.769515 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:37:23 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224641, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:37:24.129319 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:37:24.129666 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:37:24.129698 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:37:24.133983 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11531
I20260924 04:37:24.134105 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60069
Sep 24, 2026 4:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:37:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:37:26.419858 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.136s user 0.007s sys 0.156s
I20260924 04:37:26.419986 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:37:26.420090 392 mini_ranger_kms.cc:401] 127.0.98.20:60069/kms/v1/keys
Sep 24 04:37:26 dist-test-slave-jlnt krb5kdc[11138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790224646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:37:26 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:37:27 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:37:29 dist-test-slave-jlnt krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:37:29.482319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36681
--ranger_config_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60069/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:37:29.590698 11582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:37:29.590974 11582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:37:29.591049 11582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:37:29.595093 11582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:37:29.595172 11582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:37:29.595208 11582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:37:29.595263 11582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:37:29.595310 11582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:37:29.600651 11582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60069/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36681
--ranger_config_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:37:29.601711 11582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:37:29.602744 11582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:37:29.612443 11588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:37:29.612437 11587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:37:29.612445 11590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:37:29.612903 11582 server_base.cc:1034] running on GCE node
I20260924 04:37:29.613545 11582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:37:29.614889 11582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:37:29.616087 11582 hybrid_clock.cc:648] HybridClock initialized: now 1790224649616056 us; error 42 us; skew 500 ppm
Sep 24 04:37:29 dist-test-slave-jlnt krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:37:29.619554 11582 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:37:29.620447 11582 webserver.cc:460] Webserver started at http://127.0.98.62:32925/ using document root <none> and password file <none>
I20260924 04:37:29.620980 11582 fs_manager.cc:359] Metadata directory not provided
I20260924 04:37:29.621057 11582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:37:29.621309 11582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:37:29 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:37:29.679284 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d"
format_stamp: "Formatted at 2026-09-24 04:37:29 on dist-test-slave-jlnt"
server_key: "64a2669fc96f6c7a792ba31312234aaa"
server_key_iv: "67aee78368aea978d0dac372b759a205"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:29.679888 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d"
format_stamp: "Formatted at 2026-09-24 04:37:29 on dist-test-slave-jlnt"
server_key: "64a2669fc96f6c7a792ba31312234aaa"
server_key_iv: "67aee78368aea978d0dac372b759a205"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:29.683799 11582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:37:29.700001 11597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:37:29.701444 11582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 04:37:29.701577 11582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d"
format_stamp: "Formatted at 2026-09-24 04:37:29 on dist-test-slave-jlnt"
server_key: "64a2669fc96f6c7a792ba31312234aaa"
server_key_iv: "67aee78368aea978d0dac372b759a205"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:29.701689 11582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:37:29.732715 11582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:37:29.735870 11582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:37:29.736083 11582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:37:29.744433 11582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36681
I20260924 04:37:29.744431 11649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36681 every 8 connection(s)
I20260924 04:37:29.745458 11582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:37:29.748821 11650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:37:29.749451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 11582
I20260924 04:37:29.749601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:37:29.749898 392 external_mini_cluster.cc:1428] Setting key 4e884cb5e34546505301893938096080
I20260924 04:37:29.755765 11650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d: Bootstrap starting.
I20260924 04:37:29.758386 11650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d: Neither blocks nor log segments found. Creating new log.
Sep 24 04:37:29 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:37:29.759233 11650 log.cc:824] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:37:29.761631 11650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d: No bootstrap required, opened a new log
I20260924 04:37:29.765479 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:29.751469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.0.98.62:36681)
0924 04:37:29.751892 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:29.751914 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:29.751949 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:37:29.752584 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:29.752600 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:29.752919 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:29.753291 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:37:29.754169 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:29.755175 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:29.756112 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:29.756585 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:29.756750 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:29.759542 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:37:29.759548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:37:29.759561 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:37:29.759611 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:37:29.762259 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:29.762781 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:29.762786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:29.762792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:29.762906 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:29.763275 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:29.763278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:29.763280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:29.763782 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:37:29.763978 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:29.764265 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 04:37:29.764529 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":149,"threads_started":1}
I20260924 04:37:29.765599 11650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } }
I20260924 04:37:29.765883 11650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:37:29.765956 11650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c4868c9c6e4bfe84a00ae9073d4b5d, State: Initialized, Role: FOLLOWER
I20260924 04:37:29.766435 11650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } }
I20260924 04:37:29.766587 11650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:37:29.766675 11650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:37:29.766780 11650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:37:29.767616 11650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } }
I20260924 04:37:29.767973 11650 leader_election.cc:302] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c4868c9c6e4bfe84a00ae9073d4b5d; no voters:
I20260924 04:37:29.768298 11650 leader_election.cc:288] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:37:29.768388 11655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:37:29.768587 11655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [term 1 LEADER]: Becoming Leader. State: Replica: 85c4868c9c6e4bfe84a00ae9073d4b5d, State: Running, Role: LEADER
I20260924 04:37:29.768929 11655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } }
I20260924 04:37:29.769546 11650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:37:29.770161 11657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c4868c9c6e4bfe84a00ae9073d4b5d. Latest consensus state: current_term: 1 leader_uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } } }
I20260924 04:37:29.770289 11657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [sys.catalog]: This master's current role is: LEADER
I20260924 04:37:29.770776 11662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:37:29.770975 11656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c4868c9c6e4bfe84a00ae9073d4b5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36681 } } }
I20260924 04:37:29.771066 11656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d [sys.catalog]: This master's current role is: LEADER
I20260924 04:37:29.774163 11662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:37:29.774560 11650 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:37:29.785157 11662 catalog_manager.cc:1347] Generated new cluster ID: 08805a8958c04d71bf27918291966c30
I20260924 04:37:29.785277 11662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:37:29.797646 11662 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:37:29.798877 11662 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:37:29.812570 11662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d: Generated new TSK 0
I20260924 04:37:29.813423 11662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:37:30.600759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36681
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:37:30.768219 11695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:37:30.768613 11695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:37:30.768740 11695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:37:30.774740 11695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:37:30.774887 11695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:37:30.775041 11695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:37:30.780557 11695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:37:30.781704 11695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:37:30.782776 11695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:37:30.793570 11703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:37:30.793573 11705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:37:30.794137 11702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:37:30.795297 11695 server_base.cc:1034] running on GCE node
I20260924 04:37:30.796139 11695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:37:30.796891 11695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:37:30.798090 11695 hybrid_clock.cc:648] HybridClock initialized: now 1790224650798066 us; error 38 us; skew 500 ppm
Sep 24 04:37:30 dist-test-slave-jlnt krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:37:30.801621 11695 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:37:30.802593 11695 webserver.cc:460] Webserver started at http://127.0.98.1:44595/ using document root <none> and password file <none>
I20260924 04:37:30.803277 11695 fs_manager.cc:359] Metadata directory not provided
I20260924 04:37:30.803364 11695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:37:30.803601 11695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:37:30 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:37:30.820055 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "209302916fa64afc9193bc761c8eb27e"
format_stamp: "Formatted at 2026-09-24 04:37:30 on dist-test-slave-jlnt"
server_key: "86fc60e390623729c61becf469482207"
server_key_iv: "fc257b4483a45491f91697a53b8ed748"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:30.820792 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "209302916fa64afc9193bc761c8eb27e"
format_stamp: "Formatted at 2026-09-24 04:37:30 on dist-test-slave-jlnt"
server_key: "86fc60e390623729c61becf469482207"
server_key_iv: "fc257b4483a45491f91697a53b8ed748"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:30.825871 11695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:37:30.846534 11712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:37:30.848320 11695 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:37:30.848451 11695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "209302916fa64afc9193bc761c8eb27e"
format_stamp: "Formatted at 2026-09-24 04:37:30 on dist-test-slave-jlnt"
server_key: "86fc60e390623729c61becf469482207"
server_key_iv: "fc257b4483a45491f91697a53b8ed748"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:30.848587 11695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:37:30.877802 11695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:37:30.881336 11695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:37:30.881534 11695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:37:30.882165 11695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:37:30.884390 11695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:37:30.884454 11695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:37:30.884506 11695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:37:30.884577 11695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 04:37:30.898650 11695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41693
I20260924 04:37:30.899799 11695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:37:30.902446 11827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41693 every 8 connection(s)
I20260924 04:37:30.904347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 11695
I20260924 04:37:30.904453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:37:30.904795 392 external_mini_cluster.cc:1428] Setting key acd64ac9ba481d03ec31c6de4362082d
Sep 24 04:37:30 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:37:30.922698 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:30.904128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37067 (local address 127.0.98.62:36681)
0924 04:37:30.904402 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:30.904406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:30.906824 (+ 2418us) server_negotiation.cc:407] Connection header received
0924 04:37:30.908091 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:30.908095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:30.908147 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:30.908222 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:37:30.909894 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:30.910717 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:30.912404 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:30.912741 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:30.912847 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:30.917701 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:37:30.917705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:37:30.917708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:37:30.917735 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:37:30.919626 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:30.920290 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:30.920294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:30.920296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:30.920387 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:30.920808 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:30.920812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:30.920813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:30.921032 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:37:30.921125 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:30.922381 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0924 04:37:30.922509 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:37:30.924271 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:30.904631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36681 (local address 127.0.98.1:37067)
0924 04:37:30.905193 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 04:37:30.905257 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 04:37:30.907841 (+ 2584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:37:30.908647 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:37:30.908656 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:37:30.909008 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:37:30.909731 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:30.909749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:30.910856 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:30.910859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:30.912265 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:30.912276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:30.914935 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:30.914939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:30.915133 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:30.915869 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:37:30.915889 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:37:30.917534 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:37:30.919753 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:37:30.919759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:37:30.919777 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:37:30.920121 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:37:30.920496 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:37:30.920502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:37:30.920506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:37:30.920688 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:37:30.921182 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:37:30.921189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:37:30.921616 (+ 427us) client_negotiation.cc:769] Sending connection context
0924 04:37:30.922282 (+ 666us) client_negotiation.cc:240] Negotiation successful
0924 04:37:30.922618 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":143,"threads_started":1}
I20260924 04:37:30.925951 11828 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36681
I20260924 04:37:30.926203 11828 heartbeater.cc:464] Registering TS with master...
I20260924 04:37:30.926730 11828 heartbeater.cc:511] Master 127.0.98.62:36681 requested a full tablet report, sending...
I20260924 04:37:30.928247 11614 ts_manager.cc:194] Registered new tserver with Master: 209302916fa64afc9193bc761c8eb27e (127.0.98.1:41693)
I20260924 04:37:30.929462 11614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:37:30.942581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36681
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:37:30 dist-test-slave-jlnt krb5kdc[11138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790224650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:37:31.103729 11834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:37:31.104159 11834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:37:31.104249 11834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:37:31.110976 11834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:37:31.111068 11834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:37:31.111160 11834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:37:31.119217 11834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:37:31.120635 11834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:37:31.122063 11834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:37:31.134099 11841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:37:31.134092 11840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:37:31.134526 11843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:37:31.135380 11834 server_base.cc:1034] running on GCE node
I20260924 04:37:31.135994 11834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:37:31.136801 11834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:37:31.137997 11834 hybrid_clock.cc:648] HybridClock initialized: now 1790224651137963 us; error 51 us; skew 500 ppm
Sep 24 04:37:31 dist-test-slave-jlnt krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:37:31.141572 11834 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:37:31.142630 11834 webserver.cc:460] Webserver started at http://127.0.98.2:39193/ using document root <none> and password file <none>
I20260924 04:37:31.143355 11834 fs_manager.cc:359] Metadata directory not provided
I20260924 04:37:31.143462 11834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:37:31.143716 11834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:37:31 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:37:31.159183 11834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "90462b97e3954eba87d1bf7dbb68fbaf"
format_stamp: "Formatted at 2026-09-24 04:37:31 on dist-test-slave-jlnt"
server_key: "0d8ab6e1c7447b267c1c088360b71a82"
server_key_iv: "d4066c2aa64356da41fba01f78e526e7"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:31.159976 11834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "90462b97e3954eba87d1bf7dbb68fbaf"
format_stamp: "Formatted at 2026-09-24 04:37:31 on dist-test-slave-jlnt"
server_key: "0d8ab6e1c7447b267c1c088360b71a82"
server_key_iv: "d4066c2aa64356da41fba01f78e526e7"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:31.166867 11834 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260924 04:37:31.185158 11850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:37:31.186857 11834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:37:31.186981 11834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "90462b97e3954eba87d1bf7dbb68fbaf"
format_stamp: "Formatted at 2026-09-24 04:37:31 on dist-test-slave-jlnt"
server_key: "0d8ab6e1c7447b267c1c088360b71a82"
server_key_iv: "d4066c2aa64356da41fba01f78e526e7"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:31.187111 11834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:37:31.207917 11834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:37:31.211738 11834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:37:31.211987 11834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:37:31.212687 11834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:37:31.213857 11834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:37:31.213968 11834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:37:31.214067 11834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:37:31.214147 11834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:37:31.224104 11834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44353
I20260924 04:37:31.225374 11834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:37:31.227461 11963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44353 every 8 connection(s)
I20260924 04:37:31.235555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 11834
I20260924 04:37:31.235677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:37:31.235978 392 external_mini_cluster.cc:1428] Setting key 27a09ccbed6e510c563622a94a9d30a8
Sep 24 04:37:31 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:37:31.251729 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.234226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38753 (local address 127.0.98.62:36681)
0924 04:37:31.234349 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:31.234354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:31.237603 (+ 3249us) server_negotiation.cc:407] Connection header received
0924 04:37:31.239030 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:31.239037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:31.239098 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:31.239156 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:37:31.240617 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.241531 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.243392 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.243806 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.243890 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.247304 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:37:31.247310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:37:31.247316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:37:31.247348 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:37:31.249238 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:31.249912 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:31.249916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:31.249918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:31.250005 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:31.250413 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:31.250416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:31.250418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:31.250621 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:37:31.250716 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:31.251476 (+ 760us) server_negotiation.cc:299] Negotiation successful
0924 04:37:31.251585 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:37:31.252753 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.233048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36681 (local address 127.0.98.2:38753)
0924 04:37:31.237453 (+ 4405us) negotiation.cc:107] Waiting for socket to connect
0924 04:37:31.237515 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 04:37:31.238783 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:37:31.239594 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:37:31.239601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:37:31.239871 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:37:31.240474 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.240490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.241703 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.241708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.243238 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.243252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.244095 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.244102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.244350 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.245205 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:37:31.245231 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:37:31.247091 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:37:31.249371 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:37:31.249376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:37:31.249391 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:37:31.249767 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:37:31.250136 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:37:31.250139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:37:31.250141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:37:31.250305 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:37:31.250859 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:37:31.250865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:37:31.251189 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 04:37:31.251405 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 04:37:31.251721 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4159,"thread_start_us":153,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:37:31.254024 11964 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36681
I20260924 04:37:31.254302 11964 heartbeater.cc:464] Registering TS with master...
I20260924 04:37:31.254933 11964 heartbeater.cc:511] Master 127.0.98.62:36681 requested a full tablet report, sending...
I20260924 04:37:31.256127 11614 ts_manager.cc:194] Registered new tserver with Master: 90462b97e3954eba87d1bf7dbb68fbaf (127.0.98.2:44353)
I20260924 04:37:31.256814 11614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:37:31.268676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36681
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:37:31.433897 11969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:37:31.434289 11969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:37:31.434392 11969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:37:31.440248 11969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:37:31.440428 11969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:37:31.440567 11969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:37:31.448449 11969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:37:31.449930 11969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:37:31.451213 11969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:37:31.463160 11975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:37:31.463639 11974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:37:31.463884 11977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:37:31.464989 11969 server_base.cc:1034] running on GCE node
I20260924 04:37:31.465536 11969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:37:31.466354 11969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:37:31.467550 11969 hybrid_clock.cc:648] HybridClock initialized: now 1790224651467483 us; error 87 us; skew 500 ppm
Sep 24 04:37:31 dist-test-slave-jlnt krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:37:31.471148 11969 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:37:31.472142 11969 webserver.cc:460] Webserver started at http://127.0.98.3:39891/ using document root <none> and password file <none>
I20260924 04:37:31.472801 11969 fs_manager.cc:359] Metadata directory not provided
I20260924 04:37:31.472883 11969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:37:31.473111 11969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:37:31 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:37:31.492673 11969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "698d0822a27e41c1a3824312da8ecb52"
format_stamp: "Formatted at 2026-09-24 04:37:31 on dist-test-slave-jlnt"
server_key: "72d5708710caf4676d3cf4eb165bce95"
server_key_iv: "d637f465ba931cf8e3255d8a60ef2ef0"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:31.495158 11969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "698d0822a27e41c1a3824312da8ecb52"
format_stamp: "Formatted at 2026-09-24 04:37:31 on dist-test-slave-jlnt"
server_key: "72d5708710caf4676d3cf4eb165bce95"
server_key_iv: "d637f465ba931cf8e3255d8a60ef2ef0"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:31.500428 11969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
Sep 24 04:37:31 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:37:31.528721 11984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:37:31.531373 11969 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260924 04:37:31.531569 11969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "698d0822a27e41c1a3824312da8ecb52"
format_stamp: "Formatted at 2026-09-24 04:37:31 on dist-test-slave-jlnt"
server_key: "72d5708710caf4676d3cf4eb165bce95"
server_key_iv: "d637f465ba931cf8e3255d8a60ef2ef0"
server_key_version: "kuduclusterkey@0"
I20260924 04:37:31.531788 11969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:37:31.581295 11969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:37:31.584792 11969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:37:31.585021 11969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:37:31.585714 11969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:37:31.586709 11969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:37:31.586807 11969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:37:31.586899 11969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:37:31.586958 11969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:37:31.598311 11969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41729
I20260924 04:37:31.599499 11969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:37:31.604094 12097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41729 every 8 connection(s)
I20260924 04:37:31.604382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 11969
I20260924 04:37:31.604507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:37:31.604809 392 external_mini_cluster.cc:1428] Setting key 58ff5aad3ae0de4d4716dec13c71e4bf
Sep 24 04:37:31 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:37:31.622767 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.605221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52403 (local address 127.0.98.62:36681)
0924 04:37:31.605573 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:31.605578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:31.608150 (+ 2572us) server_negotiation.cc:407] Connection header received
0924 04:37:31.609347 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:31.609349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:31.609398 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:31.609446 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:37:31.611045 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.611838 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.613473 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.613813 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.613885 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.617231 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:37:31.617234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:37:31.617237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:37:31.617262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:37:31.620431 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:31.621072 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:31.621075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:31.621077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:31.621180 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:31.621553 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:31.621557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:31.621559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:31.621771 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:37:31.621847 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:31.622482 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 04:37:31.622597 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260924 04:37:31.624502 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.603133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36681 (local address 127.0.98.3:52403)
0924 04:37:31.608009 (+ 4876us) negotiation.cc:107] Waiting for socket to connect
0924 04:37:31.608065 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:37:31.609097 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:37:31.609881 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:37:31.609890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:37:31.610237 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:37:31.610891 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.610908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.611988 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.611991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.613347 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.613356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.614096 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.614100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.614288 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.614969 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:37:31.614989 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:37:31.617028 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:37:31.620574 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:37:31.620579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:37:31.620595 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:37:31.620958 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:37:31.621284 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:37:31.621287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:37:31.621289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:37:31.621451 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:37:31.621909 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:37:31.621916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:37:31.622220 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 04:37:31.623190 (+ 970us) client_negotiation.cc:240] Negotiation successful
0924 04:37:31.623507 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4653,"thread_start_us":134,"threads_started":1}
I20260924 04:37:31.626123 12098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36681
I20260924 04:37:31.626343 12098 heartbeater.cc:464] Registering TS with master...
I20260924 04:37:31.626871 12098 heartbeater.cc:511] Master 127.0.98.62:36681 requested a full tablet report, sending...
I20260924 04:37:31.628023 11614 ts_manager.cc:194] Registered new tserver with Master: 698d0822a27e41c1a3824312da8ecb52 (127.0.98.3:41729)
I20260924 04:37:31.628711 11614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52403
I20260924 04:37:31.632264 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:37:31.644629 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.634947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.0.98.62:36681)
0924 04:37:31.635051 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:31.635055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:31.635584 (+ 529us) server_negotiation.cc:407] Connection header received
0924 04:37:31.635752 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:31.635755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:31.635799 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:31.635859 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:37:31.636936 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.637737 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.639052 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.639356 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.639425 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.641238 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:37:31.641242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:37:31.641244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:37:31.641268 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:37:31.642916 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:31.643377 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:31.643381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:31.643383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:31.643458 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:37:31.643737 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:37:31.643740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:37:31.643742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:37:31.643941 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:37:31.644039 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:31.644361 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 04:37:31.644472 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:37:31.648185 11614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:37:31.652479 11614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:37:31.686688 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.666395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41693 (local address 127.0.0.1:46692)
0924 04:37:31.669781 (+ 3386us) negotiation.cc:107] Waiting for socket to connect
0924 04:37:31.669794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:37:31.669894 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:37:31.671499 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:37:31.671505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:37:31.671529 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:37:31.671674 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.671682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.681368 (+ 9686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.681371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.683226 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.683236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.686115 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.686119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.686240 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.686287 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 04:37:31.686400 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 04:37:31.686523 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3328,"thread_start_us":94,"threads_started":1}
I20260924 04:37:31.689610 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.667432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44353 (local address 127.0.0.1:33878)
0924 04:37:31.671890 (+ 4458us) negotiation.cc:107] Waiting for socket to connect
0924 04:37:31.671903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:37:31.671981 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:37:31.673411 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:37:31.673413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:37:31.673425 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:37:31.673483 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.673487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.675843 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.675846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.677273 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.677283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.689208 (+ 11925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.689212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.689304 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.689328 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:37:31.689380 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 04:37:31.689447 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4401,"thread_start_us":90,"threads_started":1}
I20260924 04:37:31.691040 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.664132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.0.1:50676)
0924 04:37:31.669114 (+ 4982us) negotiation.cc:107] Waiting for socket to connect
0924 04:37:31.669209 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0924 04:37:31.669334 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:37:31.673214 (+ 3880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:37:31.673217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:37:31.673233 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:37:31.673327 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.673333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.683560 (+ 10227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.683563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.684741 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:31.684752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.690631 (+ 5879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.690634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:31.690733 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.690754 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:37:31.690857 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:37:31.690924 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4807,"thread_start_us":84,"threads_started":1}
I20260924 04:37:31.691848 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.667831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.0.98.2:44353)
0924 04:37:31.670142 (+ 2311us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:31.670204 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:31.672072 (+ 1868us) server_negotiation.cc:407] Connection header received
0924 04:37:31.672151 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:31.672156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:31.672354 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:31.672497 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:37:31.674612 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.675690 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.680529 (+ 4839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.688900 (+ 8371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.689004 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.691519 (+ 2515us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:31.691584 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 04:37:31.691700 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2208,"thread_start_us":101,"threads_started":1}
I20260924 04:37:31.692384 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.666775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.0.98.1:41693)
0924 04:37:31.669645 (+ 2870us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:31.669653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:31.670283 (+ 630us) server_negotiation.cc:407] Connection header received
0924 04:37:31.670374 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:31.670379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:31.670600 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:31.673606 (+ 3006us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:37:31.673747 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.681231 (+ 7484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.684901 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.685780 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.685896 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.686055 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:31.692090 (+ 6035us) server_negotiation.cc:299] Negotiation successful
0924 04:37:31.692251 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2738,"thread_start_us":109,"threads_started":1}
I20260924 04:37:31.692874 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:31.665026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.0.98.3:41729)
0924 04:37:31.672780 (+ 7754us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:31.672791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:31.672822 (+ 31us) server_negotiation.cc:407] Connection header received
0924 04:37:31.672889 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:31.672894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:31.673057 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:31.677464 (+ 4407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:37:31.677656 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.683444 (+ 5788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.687073 (+ 3629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:31.690326 (+ 3253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:31.690431 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:31.690567 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:31.692615 (+ 2048us) server_negotiation.cc:299] Negotiation successful
0924 04:37:31.692725 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7650,"thread_start_us":136,"threads_started":1}
I20260924 04:37:31.696367 11898 tablet_service.cc:1467] Processing CreateTablet for tablet 8173232c7b374e90ad687d99e65da1f0 (DEFAULT_TABLE table=test-table [id=8013b71417d94e09b716d6db042c8f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:37:31.698515 11898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8173232c7b374e90ad687d99e65da1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:37:31.707741 11752 tablet_service.cc:1467] Processing CreateTablet for tablet 8173232c7b374e90ad687d99e65da1f0 (DEFAULT_TABLE table=test-table [id=8013b71417d94e09b716d6db042c8f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:37:31.708931 11752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8173232c7b374e90ad687d99e65da1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:37:31.720288 12016 tablet_service.cc:1467] Processing CreateTablet for tablet 8173232c7b374e90ad687d99e65da1f0 (DEFAULT_TABLE table=test-table [id=8013b71417d94e09b716d6db042c8f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:37:31.722679 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8173232c7b374e90ad687d99e65da1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:37:31.738062 12119 tablet_bootstrap.cc:492] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e: Bootstrap starting.
I20260924 04:37:31.740787 12119 tablet_bootstrap.cc:654] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e: Neither blocks nor log segments found. Creating new log.
I20260924 04:37:31.741575 12117 tablet_bootstrap.cc:492] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: Bootstrap starting.
I20260924 04:37:31.741997 12119 log.cc:824] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:37:31.744725 12119 tablet_bootstrap.cc:492] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e: No bootstrap required, opened a new log
I20260924 04:37:31.744925 12119 ts_tablet_manager.cc:1397] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260924 04:37:31.749357 12119 raft_consensus.cc:348] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } }
I20260924 04:37:31.750830 12119 raft_consensus.cc:374] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:37:31.750927 12119 raft_consensus.cc:729] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209302916fa64afc9193bc761c8eb27e, State: Initialized, Role: FOLLOWER
I20260924 04:37:31.751474 12119 consensus_queue.cc:260] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } }
I20260924 04:37:31.752612 12119 ts_tablet_manager.cc:1428] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 04:37:31.753034 11828 heartbeater.cc:503] Master 127.0.98.62:36681 was elected leader, sending a full tablet report...
I20260924 04:37:31.753965 12118 tablet_bootstrap.cc:492] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: Bootstrap starting.
I20260924 04:37:31.756460 12118 tablet_bootstrap.cc:654] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: Neither blocks nor log segments found. Creating new log.
I20260924 04:37:31.757882 12118 log.cc:824] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: Log is configured to *not* fsync() on all Append() calls
I20260924 04:37:31.760250 12118 tablet_bootstrap.cc:492] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: No bootstrap required, opened a new log
I20260924 04:37:31.760486 12118 ts_tablet_manager.cc:1397] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260924 04:37:31.764362 12118 raft_consensus.cc:348] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } }
I20260924 04:37:31.764708 12118 raft_consensus.cc:374] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:37:31.764808 12118 raft_consensus.cc:729] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90462b97e3954eba87d1bf7dbb68fbaf, State: Initialized, Role: FOLLOWER
I20260924 04:37:31.765405 12118 consensus_queue.cc:260] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } }
I20260924 04:37:31.766525 12118 ts_tablet_manager.cc:1428] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:37:31.766928 11964 heartbeater.cc:503] Master 127.0.98.62:36681 was elected leader, sending a full tablet report...
I20260924 04:37:31.771070 12117 tablet_bootstrap.cc:654] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: Neither blocks nor log segments found. Creating new log.
I20260924 04:37:31.772140 12117 log.cc:824] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: Log is configured to *not* fsync() on all Append() calls
I20260924 04:37:31.774500 12117 tablet_bootstrap.cc:492] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: No bootstrap required, opened a new log
I20260924 04:37:31.774710 12117 ts_tablet_manager.cc:1397] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.002s
I20260924 04:37:31.778944 12117 raft_consensus.cc:348] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } }
I20260924 04:37:31.779302 12117 raft_consensus.cc:374] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:37:31.779400 12117 raft_consensus.cc:729] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698d0822a27e41c1a3824312da8ecb52, State: Initialized, Role: FOLLOWER
I20260924 04:37:31.779999 12117 consensus_queue.cc:260] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } }
I20260924 04:37:31.781165 12117 ts_tablet_manager.cc:1428] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 04:37:31.781550 12098 heartbeater.cc:503] Master 127.0.98.62:36681 was elected leader, sending a full tablet report...
W20260924 04:37:31.852280 12099 tablet.cc:2369] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:37:31.904547 11829 tablet.cc:2369] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:37:31 dist-test-slave-jlnt krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:37:31.985333 11965 tablet.cc:2369] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:37:32.020275 12121 raft_consensus.cc:482] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:37:32.020525 12121 raft_consensus.cc:504] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } }
I20260924 04:37:32.022022 12121 leader_election.cc:288] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 698d0822a27e41c1a3824312da8ecb52 (127.0.98.3:41729), 90462b97e3954eba87d1bf7dbb68fbaf (127.0.98.2:44353)
I20260924 04:37:32.027552 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:32.022665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36949 (local address 127.0.98.2:44353)
0924 04:37:32.022945 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:32.022950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:32.023188 (+ 238us) server_negotiation.cc:407] Connection header received
0924 04:37:32.023365 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:32.023368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:32.023416 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:32.023463 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:37:32.023839 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.024649 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.026311 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.026892 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.026961 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:32.027024 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:32.027350 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 04:37:32.027422 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260924 04:37:32.028170 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:32.022287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44353 (local address 127.0.98.1:36949)
0924 04:37:32.023107 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 04:37:32.023118 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:37:32.023251 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:37:32.023602 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:37:32.023605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:37:32.023621 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:37:32.023719 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:32.023725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.024799 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.024802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:32.026152 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:32.026163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.027115 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.027118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:32.027219 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:32.027241 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:37:32.027943 (+ 702us) client_negotiation.cc:240] Negotiation successful
0924 04:37:32.028014 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":91,"threads_started":1}
I20260924 04:37:32.028930 11915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8173232c7b374e90ad687d99e65da1f0" candidate_uuid: "209302916fa64afc9193bc761c8eb27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" is_pre_election: true
I20260924 04:37:32.029371 11915 raft_consensus.cc:2454] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209302916fa64afc9193bc761c8eb27e in term 0.
I20260924 04:37:32.029943 11715 leader_election.cc:302] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209302916fa64afc9193bc761c8eb27e, 90462b97e3954eba87d1bf7dbb68fbaf; no voters:
I20260924 04:37:32.030264 12121 raft_consensus.cc:2781] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:37:32.030339 12121 raft_consensus.cc:482] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:37:32.030368 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:32.022812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42761 (local address 127.0.98.3:41729)
0924 04:37:32.023047 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:32.023053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:32.023067 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:37:32.023125 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:32.023129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:32.023184 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:32.023257 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:37:32.026885 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.027568 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.029036 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.029656 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.029750 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:32.029818 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:32.030161 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 04:37:32.030242 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260924 04:37:32.030382 12121 raft_consensus.cc:3037] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:37:32.030782 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:32.022247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.98.1:42761)
0924 04:37:32.022796 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 04:37:32.022811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:37:32.022928 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:37:32.026599 (+ 3671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:37:32.026603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:37:32.026622 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:37:32.026744 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:32.026750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.027745 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.027749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:32.028894 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:37:32.028903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.029903 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.029906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:37:32.030016 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:32.030042 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:37:32.030543 (+ 501us) client_negotiation.cc:240] Negotiation successful
0924 04:37:32.030635 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":126,"threads_started":1}
I20260924 04:37:32.031705 12121 raft_consensus.cc:504] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } }
I20260924 04:37:32.032035 12121 leader_election.cc:288] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [CANDIDATE]: Term 1 election: Requested vote from peers 698d0822a27e41c1a3824312da8ecb52 (127.0.98.3:41729), 90462b97e3954eba87d1bf7dbb68fbaf (127.0.98.2:44353)
I20260924 04:37:32.032827 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8173232c7b374e90ad687d99e65da1f0" candidate_uuid: "209302916fa64afc9193bc761c8eb27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698d0822a27e41c1a3824312da8ecb52" is_pre_election: true
I20260924 04:37:32.033103 12052 raft_consensus.cc:2454] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209302916fa64afc9193bc761c8eb27e in term 0.
I20260924 04:37:32.033427 12050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8173232c7b374e90ad687d99e65da1f0" candidate_uuid: "209302916fa64afc9193bc761c8eb27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698d0822a27e41c1a3824312da8ecb52"
I20260924 04:37:32.033524 12050 raft_consensus.cc:3037] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:37:32.034772 12050 raft_consensus.cc:2454] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209302916fa64afc9193bc761c8eb27e in term 1.
I20260924 04:37:32.036355 11716 leader_election.cc:302] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209302916fa64afc9193bc761c8eb27e, 698d0822a27e41c1a3824312da8ecb52; no voters:
I20260924 04:37:32.036840 12121 raft_consensus.cc:2781] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:37:32.036943 11915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8173232c7b374e90ad687d99e65da1f0" candidate_uuid: "209302916fa64afc9193bc761c8eb27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90462b97e3954eba87d1bf7dbb68fbaf"
I20260924 04:37:32.037070 12121 raft_consensus.cc:686] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [term 1 LEADER]: Becoming Leader. State: Replica: 209302916fa64afc9193bc761c8eb27e, State: Running, Role: LEADER
I20260924 04:37:32.037093 11915 raft_consensus.cc:3037] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:37:32.037449 12121 consensus_queue.cc:237] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } }
I20260924 04:37:32.038326 11915 raft_consensus.cc:2454] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209302916fa64afc9193bc761c8eb27e in term 1.
I20260924 04:37:32.040800 11614 catalog_manager.cc:5495] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e reported cstate change: term changed from 0 to 1, leader changed from <none> to 209302916fa64afc9193bc761c8eb27e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "209302916fa64afc9193bc761c8eb27e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "209302916fa64afc9193bc761c8eb27e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 } health_report { overall_health: UNKNOWN } } }
I20260924 04:37:32.134021 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:32.128885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.0.98.1:41693)
0924 04:37:32.129249 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0924 04:37:32.129254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:37:32.129270 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:37:32.129357 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:37:32.129360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:37:32.129432 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:37:32.129549 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:37:32.129867 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.130596 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.131894 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:37:32.132249 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:37:32.132347 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:37:32.132781 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:37:32.133360 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:37:32.133474 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:37:32.133804 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 04:37:32.133863 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260924 04:37:32.142104 11915 raft_consensus.cc:1261] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [term 1 FOLLOWER]: Refusing update from remote peer 209302916fa64afc9193bc761c8eb27e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:37:32.142424 12050 raft_consensus.cc:1261] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [term 1 FOLLOWER]: Refusing update from remote peer 209302916fa64afc9193bc761c8eb27e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:37:32.142930 12130 consensus_queue.cc:1035] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [LEADER]: Connected to new peer: Peer: permanent_uuid: "90462b97e3954eba87d1bf7dbb68fbaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:37:32.143086 12121 consensus_queue.cc:1035] T 8173232c7b374e90ad687d99e65da1f0 P 209302916fa64afc9193bc761c8eb27e [LEADER]: Connected to new peer: Peer: permanent_uuid: "698d0822a27e41c1a3824312da8ecb52" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:37:32.155871 12134 mvcc.cc:205] Tried to move back new op lower bound from 7332760175166107648 to 7332760174748860416. Current Snapshot: MvccSnapshot[applied={T|T < 7332760175166107648}]
I20260924 04:37:32.158491 12135 mvcc.cc:205] Tried to move back new op lower bound from 7332760175166107648 to 7332760174748860416. Current Snapshot: MvccSnapshot[applied={T|T < 7332760175166107648}]
I20260924 04:37:32.166695 11613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:37:32.166872 11613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:37:32.169303 11613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85c4868c9c6e4bfe84a00ae9073d4b5d: Sending DeleteTablet for 3 replicas of tablet 8173232c7b374e90ad687d99e65da1f0
I20260924 04:37:32.170197 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8173232c7b374e90ad687d99e65da1f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:37:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50676
I20260924 04:37:32.170287 11752 tablet_service.cc:1514] Processing DeleteTablet for tablet 8173232c7b374e90ad687d99e65da1f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:37:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46692
I20260924 04:37:32.170320 11898 tablet_service.cc:1514] Processing DeleteTablet for tablet 8173232c7b374e90ad687d99e65da1f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:37:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33878
I20260924 04:37:32.171044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 11695
I20260924 04:37:32.172151 12147 tablet_replica.cc:330] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: stopping tablet replica
I20260924 04:37:32.172386 12149 tablet_replica.cc:330] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: stopping tablet replica
I20260924 04:37:32.172750 12149 raft_consensus.cc:2229] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:37:32.173099 12149 raft_consensus.cc:2258] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:37:32.173497 12147 raft_consensus.cc:2229] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:37:32.173830 12147 raft_consensus.cc:2258] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:37:32.175184 12147 ts_tablet_manager.cc:1905] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:37:32.175185 12149 ts_tablet_manager.cc:1905] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:37:32.177940 12149 ts_tablet_manager.cc:1918] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:37:32.177940 12147 ts_tablet_manager.cc:1918] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:37:32.178040 12147 log.cc:1196] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/8173232c7b374e90ad687d99e65da1f0
I20260924 04:37:32.178148 12149 log.cc:1196] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/8173232c7b374e90ad687d99e65da1f0
I20260924 04:37:32.178361 12147 ts_tablet_manager.cc:1939] T 8173232c7b374e90ad687d99e65da1f0 P 698d0822a27e41c1a3824312da8ecb52: Deleting consensus metadata
I20260924 04:37:32.178709 12149 ts_tablet_manager.cc:1939] T 8173232c7b374e90ad687d99e65da1f0 P 90462b97e3954eba87d1bf7dbb68fbaf: Deleting consensus metadata
I20260924 04:37:32.179764 11600 catalog_manager.cc:4841] TS 90462b97e3954eba87d1bf7dbb68fbaf (127.0.98.2:44353): tablet 8173232c7b374e90ad687d99e65da1f0 (table test-table [id=8013b71417d94e09b716d6db042c8f2f]) successfully deleted
I20260924 04:37:32.179804 11601 catalog_manager.cc:4841] TS 698d0822a27e41c1a3824312da8ecb52 (127.0.98.3:41729): tablet 8173232c7b374e90ad687d99e65da1f0 (table test-table [id=8013b71417d94e09b716d6db042c8f2f]) successfully deleted
W20260924 04:37:32.180753 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41693 (error 108)
I20260924 04:37:32.181605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 11834
I20260924 04:37:32.181802 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:37:32.181458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41693 (local address 127.0.0.1:46712)
0924 04:37:32.181604 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:37:32.181674 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:37:32.183334 11599 catalog_manager.cc:4584] TS 209302916fa64afc9193bc761c8eb27e (127.0.98.1:41693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8173232c7b374e90ad687d99e65da1f0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41693: connect: Connection refused (error 111)
I20260924 04:37:32.187461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 11969
I20260924 04:37:32.194921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 11582
I20260924 04:37:32.205040 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:37:32.539399 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:37:32.539546 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:37:32.883957 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:37:32.884 UTC [11168] LOG: received smart shutdown request
2026-09-24 04:37:32.887 UTC [11168] LOG: background worker "logical replication launcher" (PID 11177) exited with exit code 1
2026-09-24 04:37:32.888 UTC [11172] LOG: shutting down
2026-09-24 04:37:33.061 UTC [11168] LOG: database system is shut down
2026-09-24T04:37:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46321 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:37:33 dist-test-slave-jlnt krb5kdc[12162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:37:33 dist-test-slave-jlnt krb5kdc[12162](info): set up 2 sockets
Sep 24 04:37:33 dist-test-slave-jlnt krb5kdc[12162](info): commencing operation
krb5kdc: starting...
W20260924 04:37:34.225219 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:37:34 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:37:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:37:34.392184 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVN1buQ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:37:36.856 UTC [12192] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:37:36.856 UTC [12192] LOG: listening on IPv4 address "127.0.98.20", port 57127
2026-09-24 04:37:36.859 UTC [12192] LOG: listening on Unix socket "/tmp/.s.PGSQL.57127"
2026-09-24 04:37:36.880 UTC [12195] LOG: database system was shut down at 2026-09-24 04:37:35 UTC
2026-09-24 04:37:36.884 UTC [12192] LOG: database system is ready to accept connections
I20260924 04:37:37.837065 392 mini_postgres.cc:96] Postgres bound to 57127
2026-09-24 04:37:37.843 UTC [12203] FATAL: database "slave" does not exist
127.0.98.20:57127 - accepting connections
I20260924 04:37:37.843780 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:37:37.858474 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:37:38.311966 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:37:38.312048 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-admin
2026-09-24 04:37:38,587 [I] DB FLAVOR :POSTGRES
2026-09-24 04:37:38,588 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:37:38,588 [I] Checking connection..
2026-09-24 04:37:38,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:37:38,836 [I] Checking connection passed.
2026-09-24 04:37:38,837 [I] --------- Verifying version history table ---------
2026-09-24 04:37:38,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:37:39,093 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:37:39,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:37:39,347 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:37:39,347 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:37:39,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:37:39.597 UTC [12267] WARNING: there is no transaction in progress
2026-09-24 04:37:39,603 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:37:39,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:37:39,844 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:37:39,845 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:37:39,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:37:40,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:37:40,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jlnt.c.gcp-upstream.internal','N') ;"
2026-09-24 04:37:40,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:37:40,776 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:37:40,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:37:41,005 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:37:41,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:37:41,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:37:41,479 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:37:41,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:37:42.411 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.427 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.443 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.458 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.473 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.492 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.553 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.560 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.575 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.584 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.594 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.608 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.617 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.623 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.631 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:42.637 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:37:43,253 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:37:43,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:37:43,500 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:37:43,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:37:43,750 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:37:43,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:37:43,973 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:37:43,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:37:44,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:37:44,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jlnt.c.gcp-upstream.internal';"
2026-09-24 04:37:44,683 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:37:44,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:37:44,935 [I] DB_PATCHES have already been applied
I20260924 04:37:44.941218 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVN1buQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:37:44.941274 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:37:44.944541 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59571
Sep 24, 2026 4:37:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:37:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:37:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:37:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:37:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:37:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:37:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:37:46.042304 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.199s user 0.000s sys 0.009s
Sep 24, 2026 4:38:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:38:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:38:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:38:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:38:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:38:06.461463 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:38:06 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224686, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:38:06.545042 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:38:06.553542 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:38:07.430626 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:38:07.430709 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-kms
I20260924 04:38:07.430727 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:57127
2026-09-24 04:38:07,599 [I] DB FLAVOR :POSTGRES
2026-09-24 04:38:07,599 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:38:07,599 [I] Checking connection
2026-09-24 04:38:07,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:38:07,819 [I] connection success
2026-09-24 04:38:07,819 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:38:07,819 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:38:07,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:38:08,065 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:38:08,066 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:38:08,066 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:38:08,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57127/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:38:08,326 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:38:08.759518 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:38:08.948877 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:38:08.994262 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:38:08 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224686, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:09.380918 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:38:09.381052 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVN1buQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVN1buQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:38:09.381072 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:38:09.384706 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12555
I20260924 04:38:09.384788 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:41239
Sep 24, 2026 4:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:38:11.579056 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.034s user 0.000s sys 0.151s
I20260924 04:38:11.579190 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:38:11.579300 392 mini_ranger_kms.cc:401] 127.0.98.20:41239/kms/v1/keys
Sep 24 04:38:11 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790224691, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:38:12 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224691, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:38:12 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224691, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:38:14 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:38:14.569043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33265
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33265
--ranger_config_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41239/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:14.677637 12606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:14.677866 12606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:14.677904 12606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:14.681862 12606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:14.681913 12606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:14.681932 12606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:14.681948 12606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:14.681988 12606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:14.687382 12606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:41239/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33265
--ranger_config_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:14.688375 12606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:14.689373 12606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:14.697999 12611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:14.698091 12612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:14.698217 12614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:14.698087 12606 server_base.cc:1034] running on GCE node
I20260924 04:38:14.698891 12606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:14.699905 12606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:14.701115 12606 hybrid_clock.cc:648] HybridClock initialized: now 1790224694701091 us; error 42 us; skew 500 ppm
Sep 24 04:38:14 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:14.714439 12606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:38:14.715374 12606 webserver.cc:460] Webserver started at http://127.0.98.62:40667/ using document root <none> and password file <none>
I20260924 04:38:14.715873 12606 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:14.715927 12606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:14.716086 12606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:38:14 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:14.762604 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "ccbf8d08002449c1848c5de76bfae585"
format_stamp: "Formatted at 2026-09-24 04:38:14 on dist-test-slave-jlnt"
server_key: "5f7ca7a4dcc5b381cf6bd2f1599ae5be"
server_key_iv: "c617c38319b22515daca812e5030a560"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:14.763229 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "ccbf8d08002449c1848c5de76bfae585"
format_stamp: "Formatted at 2026-09-24 04:38:14 on dist-test-slave-jlnt"
server_key: "5f7ca7a4dcc5b381cf6bd2f1599ae5be"
server_key_iv: "c617c38319b22515daca812e5030a560"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:14.767246 12606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 04:38:14.781091 12621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:14.782516 12606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:38:14.782657 12606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "ccbf8d08002449c1848c5de76bfae585"
format_stamp: "Formatted at 2026-09-24 04:38:14 on dist-test-slave-jlnt"
server_key: "5f7ca7a4dcc5b381cf6bd2f1599ae5be"
server_key_iv: "c617c38319b22515daca812e5030a560"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:14.782800 12606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:14.799074 12606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:14.801844 12606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:14.802037 12606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:14.810117 12606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33265
I20260924 04:38:14.810127 12673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33265 every 8 connection(s)
I20260924 04:38:14.811034 12606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:38:14.813743 12674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:14.815925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 12606
I20260924 04:38:14.816056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:38:14.816370 392 external_mini_cluster.cc:1428] Setting key 75568d8ef6ef99abe541f8db73b0cf94
I20260924 04:38:14.819779 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Bootstrap starting.
I20260924 04:38:14.822250 12674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:14.823175 12674 log.cc:824] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:14.825225 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: No bootstrap required, opened a new log
Sep 24 04:38:14 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:14.829393 12674 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } }
I20260924 04:38:14.829640 12674 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:14.829718 12674 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbf8d08002449c1848c5de76bfae585, State: Initialized, Role: FOLLOWER
I20260924 04:38:14.830211 12674 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } }
I20260924 04:38:14.830384 12674 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:38:14.830500 12674 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:38:14.830619 12674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:14.831533 12674 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } }
I20260924 04:38:14.831923 12674 leader_election.cc:302] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccbf8d08002449c1848c5de76bfae585; no voters:
I20260924 04:38:14.832253 12674 leader_election.cc:288] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:38:14.832317 12679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:38:14.832623 12679 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 1 LEADER]: Becoming Leader. State: Replica: ccbf8d08002449c1848c5de76bfae585, State: Running, Role: LEADER
I20260924 04:38:14.832959 12679 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } }
I20260924 04:38:14.833039 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:14.817777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.0.98.62:33265)
0924 04:38:14.818220 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:14.818238 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:14.818279 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:38:14.819113 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:14.819136 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:14.819560 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:14.819944 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:14.820824 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:14.822373 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:14.823556 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:14.824272 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:14.824435 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:14.827598 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:14.827605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:14.827620 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:14.827650 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:14.829726 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:14.830217 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:14.830223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:14.830229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:14.830364 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:14.830676 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:14.830679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:14.830681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:14.831169 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:14.831383 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:14.831676 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 04:38:14.832111 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
I20260924 04:38:14.833616 12674 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:14.834167 12679 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbf8d08002449c1848c5de76bfae585. Latest consensus state: current_term: 1 leader_uuid: "ccbf8d08002449c1848c5de76bfae585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } } }
I20260924 04:38:14.834275 12679 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:14.834465 12679 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccbf8d08002449c1848c5de76bfae585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } } }
I20260924 04:38:14.834546 12679 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:14.835749 12674 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:38:14.836241 12685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:38:14.842355 12685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:38:14.849660 12685 catalog_manager.cc:1347] Generated new cluster ID: 7c2104486d8047c199b23e6c706f82aa
I20260924 04:38:14.849720 12685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:38:14.863229 12685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:38:14.864504 12685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:38:14.877058 12685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Generated new TSK 0
I20260924 04:38:14.877873 12685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:38:15.576862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33265
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:15.692701 12719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:15.692955 12719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:15.693038 12719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:15.697227 12719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:15.697294 12719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:15.697391 12719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:38:15.702478 12719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:41239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:15.703404 12719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:15.704342 12719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:15.714736 12727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:15.714742 12725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:15.714742 12724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:15.715054 12719 server_base.cc:1034] running on GCE node
I20260924 04:38:15.715674 12719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:15.716408 12719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:15.717573 12719 hybrid_clock.cc:648] HybridClock initialized: now 1790224695717542 us; error 51 us; skew 500 ppm
Sep 24 04:38:15 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:15.720289 12719 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:38:15.721359 12719 webserver.cc:460] Webserver started at http://127.0.98.1:46261/ using document root <none> and password file <none>
I20260924 04:38:15.721975 12719 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:15.722061 12719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:15.722289 12719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:38:15 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:15.738602 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "42f9fe2bcacb441a817157b3097a22d2"
format_stamp: "Formatted at 2026-09-24 04:38:15 on dist-test-slave-jlnt"
server_key: "95b57bc9bd1245fb4b6c09ab13a15e27"
server_key_iv: "e70698c3a3e0fad72e36b70a1e0d9bb6"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:15.739320 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "42f9fe2bcacb441a817157b3097a22d2"
format_stamp: "Formatted at 2026-09-24 04:38:15 on dist-test-slave-jlnt"
server_key: "95b57bc9bd1245fb4b6c09ab13a15e27"
server_key_iv: "e70698c3a3e0fad72e36b70a1e0d9bb6"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:15.744355 12719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 04:38:15.760530 12736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:15.762220 12719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:38:15.762370 12719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "42f9fe2bcacb441a817157b3097a22d2"
format_stamp: "Formatted at 2026-09-24 04:38:15 on dist-test-slave-jlnt"
server_key: "95b57bc9bd1245fb4b6c09ab13a15e27"
server_key_iv: "e70698c3a3e0fad72e36b70a1e0d9bb6"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:15.762498 12719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:15.791821 12719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:15.795466 12719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:15.795707 12719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:15.796424 12719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:15.797610 12719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:15.797716 12719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:15.797840 12719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:15.797926 12719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:15.808395 12719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39925
I20260924 04:38:15.811661 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39925 every 8 connection(s)
I20260924 04:38:15.812314 12719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:38:15.813462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 12719
I20260924 04:38:15.813586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:38:15.813882 392 external_mini_cluster.cc:1428] Setting key bf9f51e397386fd161462381398b740d
Sep 24 04:38:15 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:38:15.835826 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:15.815797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33265 (local address 127.0.98.1:53301)
0924 04:38:15.816352 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:15.816410 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:15.817448 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:15.819434 (+ 1986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:15.819443 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:15.819767 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:15.820376 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:15.820392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:15.821609 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:15.821613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:15.823001 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:15.823012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:15.823676 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:15.823680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:15.823884 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:15.824591 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:15.824613 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:15.826338 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:15.830294 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:15.830302 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:15.830318 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:15.830656 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:15.830991 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:15.830994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:15.830996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:15.831184 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:15.833681 (+ 2497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:15.833687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:15.834142 (+ 455us) client_negotiation.cc:769] Sending connection context
0924 04:38:15.834395 (+ 253us) client_negotiation.cc:240] Negotiation successful
0924 04:38:15.834808 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:38:15.838423 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:15.816792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53301 (local address 127.0.98.62:33265)
0924 04:38:15.818942 (+ 2150us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:15.818952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:15.818967 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:38:15.819006 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:15.819008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:15.819049 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:15.819137 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:15.820542 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:15.821454 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:15.823139 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:15.823505 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:15.823622 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:15.826528 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:15.826531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:15.826535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:15.826561 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:15.830125 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:15.830776 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:15.830780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:15.830781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:15.830883 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:15.833294 (+ 2411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:15.833298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:15.833300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:15.833534 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:15.833629 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:15.838115 (+ 4486us) server_negotiation.cc:299] Negotiation successful
0924 04:38:15.838249 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2066,"thread_start_us":99,"threads_started":1}
I20260924 04:38:15.840202 12850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33265
I20260924 04:38:15.840440 12850 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:15.841034 12850 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:15.842716 12638 ts_manager.cc:194] Registered new tserver with Master: 42f9fe2bcacb441a817157b3097a22d2 (127.0.98.1:39925)
I20260924 04:38:15.844264 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:38:15.864259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33265
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:38:15 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790224695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:38:15.998126 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:15.998476 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:15.998535 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:16.004463 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:16.004546 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:16.004627 12858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:38:16.012465 12858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:41239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:16.013996 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:16.015326 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:16.027431 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:16.027436 12864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:16.027993 12865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:16.029948 12858 server_base.cc:1034] running on GCE node
I20260924 04:38:16.030550 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:16.031325 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:16.032517 12858 hybrid_clock.cc:648] HybridClock initialized: now 1790224696032445 us; error 86 us; skew 500 ppm
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:16.036106 12858 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:38:16.037053 12858 webserver.cc:460] Webserver started at http://127.0.98.2:44243/ using document root <none> and password file <none>
I20260924 04:38:16.037637 12858 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:16.037736 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:16.037961 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:16.052426 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "ccdc79361663476ba6f0364130603e5e"
format_stamp: "Formatted at 2026-09-24 04:38:16 on dist-test-slave-jlnt"
server_key: "d1b4c906aac6d59e1b583377d401834b"
server_key_iv: "e21981ba67df71450c8d481bff1a2cf2"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:16.053259 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ccdc79361663476ba6f0364130603e5e"
format_stamp: "Formatted at 2026-09-24 04:38:16 on dist-test-slave-jlnt"
server_key: "d1b4c906aac6d59e1b583377d401834b"
server_key_iv: "e21981ba67df71450c8d481bff1a2cf2"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:16.061481 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260924 04:38:16.087186 12874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:16.088970 12858 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260924 04:38:16.089113 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "ccdc79361663476ba6f0364130603e5e"
format_stamp: "Formatted at 2026-09-24 04:38:16 on dist-test-slave-jlnt"
server_key: "d1b4c906aac6d59e1b583377d401834b"
server_key_iv: "e21981ba67df71450c8d481bff1a2cf2"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:16.089279 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:16.108549 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:16.112305 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:16.112543 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:16.113281 12858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:16.114349 12858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:16.114449 12858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:16.114544 12858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:16.114602 12858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:16.125298 12858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44107
I20260924 04:38:16.126431 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:38:16.131143 12987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44107 every 8 connection(s)
I20260924 04:38:16.132057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 12858
I20260924 04:38:16.132189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:38:16.132513 392 external_mini_cluster.cc:1428] Setting key fb9ee32c80ecffb43172195dfe2ba961
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:16.162081 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:16.133396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46731 (local address 127.0.98.62:33265)
0924 04:38:16.134030 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:16.134034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:16.144500 (+ 10466us) server_negotiation.cc:407] Connection header received
0924 04:38:16.145995 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:16.145999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:16.146062 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:16.146124 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:16.149859 (+ 3735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:16.150685 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:16.152469 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:16.152821 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:16.152902 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:16.156037 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:16.156041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:16.156043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:16.156073 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:16.158148 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:16.160193 (+ 2045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:16.160198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:16.160200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:16.160301 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:16.160707 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:16.160710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:16.160712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:16.160936 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:16.161020 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:16.161752 (+ 732us) server_negotiation.cc:299] Negotiation successful
0924 04:38:16.161889 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552}
I20260924 04:38:16.164036 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:16.135098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33265 (local address 127.0.98.2:46731)
0924 04:38:16.144322 (+ 9224us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:16.144386 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:16.145597 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:16.146558 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:16.146568 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:16.147073 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:16.149677 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:16.149699 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:16.150854 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:16.150857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:16.152322 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:16.152333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:16.153113 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:16.153117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:16.153366 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:16.154080 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:16.154101 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:16.155847 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:16.158318 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:16.158324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:16.158342 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:16.158678 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:16.160418 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:16.160421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:16.160423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:16.160595 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:16.161087 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:16.161093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:16.161469 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 04:38:16.162589 (+ 1120us) client_negotiation.cc:240] Negotiation successful
0924 04:38:16.162940 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8951,"thread_start_us":156,"threads_started":1}
I20260924 04:38:16.169572 12988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33265
I20260924 04:38:16.169886 12988 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:16.170513 12988 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:16.171810 12638 ts_manager.cc:194] Registered new tserver with Master: ccdc79361663476ba6f0364130603e5e (127.0.98.2:44107)
I20260924 04:38:16.172595 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:38:16.186224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33265
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:16.343122 12993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:16.343520 12993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:16.343626 12993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:16.349521 12993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:16.349670 12993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:16.349797 12993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:38:16.357117 12993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:41239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:16.358577 12993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:16.359856 12993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:16.378777 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:16.379168 13001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:16.378841 12998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:16.379683 12993 server_base.cc:1034] running on GCE node
I20260924 04:38:16.380162 12993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:16.380950 12993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:16.382104 12993 hybrid_clock.cc:648] HybridClock initialized: now 1790224696382047 us; error 86 us; skew 500 ppm
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:16.385780 12993 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:38:16.387013 12993 webserver.cc:460] Webserver started at http://127.0.98.3:43891/ using document root <none> and password file <none>
I20260924 04:38:16.387858 12993 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:16.387997 12993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:16.388271 12993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:16.411571 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf64e9b13ebe4b0bab643bf88bef6c13"
format_stamp: "Formatted at 2026-09-24 04:38:16 on dist-test-slave-jlnt"
server_key: "62679fa2d3739b8f8dd27b24d6cd4a72"
server_key_iv: "fba0d5c8b8bd038140d0a564d09786b9"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:16.413376 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf64e9b13ebe4b0bab643bf88bef6c13"
format_stamp: "Formatted at 2026-09-24 04:38:16 on dist-test-slave-jlnt"
server_key: "62679fa2d3739b8f8dd27b24d6cd4a72"
server_key_iv: "fba0d5c8b8bd038140d0a564d09786b9"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:16.418951 12993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:38:16.444782 13008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:16.446537 12993 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:38:16.446662 12993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "bf64e9b13ebe4b0bab643bf88bef6c13"
format_stamp: "Formatted at 2026-09-24 04:38:16 on dist-test-slave-jlnt"
server_key: "62679fa2d3739b8f8dd27b24d6cd4a72"
server_key_iv: "fba0d5c8b8bd038140d0a564d09786b9"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:16.446789 12993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:16.476420 12993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:16.480116 12993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:16.480960 12993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:16.481756 12993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:16.483862 12993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:16.483968 12993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:16.484073 12993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:16.484139 12993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:16.498008 12993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33439
I20260924 04:38:16.499259 12993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:38:16.500424 13121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33439 every 8 connection(s)
I20260924 04:38:16.508082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 12993
I20260924 04:38:16.508205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:38:16.508493 392 external_mini_cluster.cc:1428] Setting key 484db588f959b1a5a7f8510efce76058
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:16.517637 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:16.502488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59617 (local address 127.0.98.62:33265)
0924 04:38:16.502613 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:16.502618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:16.504459 (+ 1841us) server_negotiation.cc:407] Connection header received
0924 04:38:16.505744 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:16.505747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:16.505803 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:16.505858 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:16.507449 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:16.508279 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:16.510058 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:16.510414 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:16.510495 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:16.513485 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:16.513489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:16.513492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:16.513518 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:16.515269 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:16.515874 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:16.515878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:16.515880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:16.515958 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:16.516357 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:16.516360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:16.516362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:16.516572 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:16.516654 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:16.517362 (+ 708us) server_negotiation.cc:299] Negotiation successful
0924 04:38:16.517478 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:38:16.519621 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:16.503028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33265 (local address 127.0.98.3:59617)
0924 04:38:16.504284 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:16.504353 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:16.505493 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:16.506259 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:16.506268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:16.506629 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:16.507269 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:16.507289 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:16.508448 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:16.508452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:16.509915 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:16.509926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:16.510689 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:16.510694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:16.510887 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:16.511524 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:16.511544 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:16.513305 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:16.515411 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:16.515417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:16.515431 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:16.515753 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:16.516060 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:16.516063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:16.516065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:16.516230 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:16.516719 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:16.516725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:16.517028 (+ 303us) client_negotiation.cc:769] Sending connection context
0924 04:38:16.518130 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0924 04:38:16.518617 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":155,"threads_started":1}
I20260924 04:38:16.521029 13122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33265
I20260924 04:38:16.521344 13122 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:16.521886 13122 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:16.522943 12638 ts_manager.cc:194] Registered new tserver with Master: bf64e9b13ebe4b0bab643bf88bef6c13 (127.0.98.3:33439)
I20260924 04:38:16.523638 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59617
I20260924 04:38:16.533378 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:38:16.534056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 12719
I20260924 04:38:16.540511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 12858
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:16.548048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 12993
I20260924 04:38:16.555757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 12606
I20260924 04:38:16.562633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33265
--webserver_interface=127.0.98.62
--webserver_port=40667
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33265
--ranger_config_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41239/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41239/kms with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:16.690169 13125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:16.690403 13125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:16.690440 13125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:16.694345 13125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:16.694389 13125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:16.694407 13125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:16.694423 13125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:16.694437 13125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:16.699574 13125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41239/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33265
--ranger_config_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:16.700419 13125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:16.701340 13125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:16.709616 13130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:16.709623 13131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:16.709674 13125 server_base.cc:1034] running on GCE node
W20260924 04:38:16.709648 13133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:16.710464 13125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:16.711498 13125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:16.712694 13125 hybrid_clock.cc:648] HybridClock initialized: now 1790224696712676 us; error 32 us; skew 500 ppm
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:16.715370 13125 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:38:16.716234 13125 webserver.cc:460] Webserver started at http://127.0.98.62:40667/ using document root <none> and password file <none>
I20260924 04:38:16.716753 13125 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:16.716845 13125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:16.777751 13125 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:38:16.779994 13141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:16.781400 13125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:38:16.781538 13125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "ccbf8d08002449c1848c5de76bfae585"
format_stamp: "Formatted at 2026-09-24 04:38:14 on dist-test-slave-jlnt"
server_key: "5f7ca7a4dcc5b381cf6bd2f1599ae5be"
server_key_iv: "c617c38319b22515daca812e5030a560"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:16.782073 13125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:16.806083 13125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:16.809011 13125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:16.809250 13125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:16.816363 13125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33265
I20260924 04:38:16.816380 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33265 every 8 connection(s)
I20260924 04:38:16.817448 13125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:38:16.820343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 13125
I20260924 04:38:16.820780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39925
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33265
--webserver_port=46261
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41239/kms with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:38:16.821648 13194 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:38:16.823215 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Bootstrap starting.
I20260924 04:38:16.838985 13194 log.cc:824] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:16.845238 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:38:16.845608 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Bootstrap complete.
I20260924 04:38:16.849570 13194 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } }
I20260924 04:38:16.849993 13194 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbf8d08002449c1848c5de76bfae585, State: Initialized, Role: FOLLOWER
I20260924 04:38:16.850445 13194 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } }
I20260924 04:38:16.850589 13194 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:38:16.850668 13194 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:38:16.850785 13194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:38:16.852545 13194 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } }
I20260924 04:38:16.852867 13194 leader_election.cc:302] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccbf8d08002449c1848c5de76bfae585; no voters:
I20260924 04:38:16.853192 13194 leader_election.cc:288] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:38:16.853273 13198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:38:16.853551 13198 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [term 2 LEADER]: Becoming Leader. State: Replica: ccbf8d08002449c1848c5de76bfae585, State: Running, Role: LEADER
I20260924 04:38:16.853892 13198 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } }
I20260924 04:38:16.854365 13194 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:16.855180 13199 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ccbf8d08002449c1848c5de76bfae585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } } }
I20260924 04:38:16.855300 13199 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:16.855654 13200 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbf8d08002449c1848c5de76bfae585. Latest consensus state: current_term: 2 leader_uuid: "ccbf8d08002449c1848c5de76bfae585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbf8d08002449c1848c5de76bfae585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33265 } } }
I20260924 04:38:16.855787 13200 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:16.856117 13207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:38:16.856557 13194 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:38:16.862198 13207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:38:16.863116 13207 catalog_manager.cc:1259] Loaded cluster ID: 7c2104486d8047c199b23e6c706f82aa
I20260924 04:38:16.863186 13207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:38:16.865989 13207 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:38:16.866804 13207 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Loaded TSK: 0
I20260924 04:38:16.868187 13207 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:38:16.957793 13196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:16.958098 13196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:16.958170 13196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:16.963963 13196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:16.964035 13196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:16.964123 13196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:38:16.971560 13196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46261
--tserver_master_addrs=127.0.98.62:33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:16.972940 13196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:16.974193 13196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:16.986191 13234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:16.986696 13196 server_base.cc:1034] running on GCE node
W20260924 04:38:16.986219 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:16.987511 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:16.988009 13196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:16.988745 13196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:16.989915 13196 hybrid_clock.cc:648] HybridClock initialized: now 1790224696989893 us; error 38 us; skew 500 ppm
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:16.993362 13196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:38:16.994494 13196 webserver.cc:460] Webserver started at http://127.0.98.1:46261/ using document root <none> and password file <none>
I20260924 04:38:16.995206 13196 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:16.995366 13196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:38:16 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:17.019191 13196 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:38:17.026621 13242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:17.028470 13196 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260924 04:38:17.028616 13196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "42f9fe2bcacb441a817157b3097a22d2"
format_stamp: "Formatted at 2026-09-24 04:38:15 on dist-test-slave-jlnt"
server_key: "95b57bc9bd1245fb4b6c09ab13a15e27"
server_key_iv: "e70698c3a3e0fad72e36b70a1e0d9bb6"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:17.029372 13196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:17.044277 13196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:17.049007 13196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:17.049306 13196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:17.050086 13196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:17.051131 13196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:17.051235 13196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:17.051362 13196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:17.051446 13196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:17.064461 13196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39925
I20260924 04:38:17.065814 13196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:38:17.068881 13355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39925 every 8 connection(s)
I20260924 04:38:17.073741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 13196
I20260924 04:38:17.074232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44107
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33265
--webserver_port=44243
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41239/kms with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:38:17 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:17.093715 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:17.073568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33265 (local address 127.0.98.1:38415)
0924 04:38:17.075166 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:17.075224 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:17.076583 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:17.078201 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:17.078208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:17.078566 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:17.079240 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:17.079262 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.081346 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.081351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:17.082971 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:17.082986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.084272 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.084279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:17.084511 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:17.085295 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:17.085321 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:17.087494 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:17.090259 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:17.090265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:17.090280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:17.090611 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:17.091016 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:17.091019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:17.091021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:17.091188 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:17.091881 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:17.091888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:17.092181 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 04:38:17.092363 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:38:17.092677 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"thread_start_us":146,"threads_started":1}
I20260924 04:38:17.095498 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:17.075049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38415 (local address 127.0.98.62:33265)
0924 04:38:17.077107 (+ 2058us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:17.077125 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:17.077207 (+ 82us) server_negotiation.cc:407] Connection header received
0924 04:38:17.077455 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:17.077465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:17.077811 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:17.078135 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:17.079442 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.081131 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.083148 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.084070 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.085100 (+ 1030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:17.087884 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:17.087893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:17.087916 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:17.087950 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:17.090122 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:17.090756 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:17.090761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:17.090767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:17.090911 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:17.091291 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:17.091294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:17.091296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:17.091740 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:17.094239 (+ 2499us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:17.094383 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 04:38:17.094656 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1893,"thread_start_us":117,"threads_started":1}
I20260924 04:38:17.097993 13356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33265
I20260924 04:38:17.098234 13356 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:17.098757 13356 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
W20260924 04:38:17.100885 13356 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33265 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:38:17.101002 13356 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:17.101173 13356 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:17.102039 13356 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:17.102193 13356 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
W20260924 04:38:17.102993 13356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:38:17.224169 13360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:17.224458 13360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:17.224532 13360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:17.229970 13360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:17.230094 13360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:17.230228 13360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:38:17.236370 13360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44243
--tserver_master_addrs=127.0.98.62:33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:17.237505 13360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:17.238580 13360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:17.248351 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:17.248389 13366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:17.248733 13360 server_base.cc:1034] running on GCE node
W20260924 04:38:17.249886 13365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:17.250388 13360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:17.251139 13360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:17.252297 13360 hybrid_clock.cc:648] HybridClock initialized: now 1790224697252272 us; error 44 us; skew 500 ppm
Sep 24 04:38:17 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:17.256660 13360 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:38:17.258280 13360 webserver.cc:460] Webserver started at http://127.0.98.2:44243/ using document root <none> and password file <none>
I20260924 04:38:17.259327 13360 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:17.259557 13360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:38:17 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:17.290992 13360 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.000s
I20260924 04:38:17.306607 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:17.308537 13360 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.001s
I20260924 04:38:17.308688 13360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "ccdc79361663476ba6f0364130603e5e"
format_stamp: "Formatted at 2026-09-24 04:38:16 on dist-test-slave-jlnt"
server_key: "d1b4c906aac6d59e1b583377d401834b"
server_key_iv: "e21981ba67df71450c8d481bff1a2cf2"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:17.309512 13360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:17.339752 13360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:17.343492 13360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:17.343706 13360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:17.344446 13360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:17.345505 13360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:17.345567 13360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:17.345623 13360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:17.345654 13360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:17.356405 13360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44107
I20260924 04:38:17.356501 13489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44107 every 8 connection(s)
I20260924 04:38:17.358457 13360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:38:17.362133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 13360
I20260924 04:38:17.362665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33439
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33265
--webserver_port=43891
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41239/kms with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:38:17 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:17.375556 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:17.360860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47223 (local address 127.0.98.62:33265)
0924 04:38:17.361007 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:17.361012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:17.361605 (+ 593us) server_negotiation.cc:407] Connection header received
0924 04:38:17.362890 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:17.362893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:17.362943 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:17.362996 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:17.364593 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.365484 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.367454 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.367885 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.367974 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:17.371173 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:17.371177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:17.371179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:17.371223 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:17.372979 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:17.373652 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:17.373656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:17.373658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:17.373751 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:17.374149 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:17.374152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:17.374154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:17.374392 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:17.374482 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:17.375230 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 04:38:17.375372 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:38:17.381950 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:17.360849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33265 (local address 127.0.98.2:47223)
0924 04:38:17.361444 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:17.361500 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:17.362623 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:17.363405 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:17.363414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:17.363761 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:17.364414 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:17.364435 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.365656 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.365661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:17.367309 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:17.367323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.368175 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.368181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:17.368413 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:17.369269 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:17.369290 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:17.370984 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:17.373116 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:17.373122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:17.373171 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:17.373531 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:17.373868 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:17.373871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:17.373873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:17.374040 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:17.374553 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:17.374560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:17.374906 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 04:38:17.375975 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0924 04:38:17.376338 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":195,"threads_started":1}
I20260924 04:38:17.383306 13490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33265
I20260924 04:38:17.383540 13490 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:17.384088 13490 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
W20260924 04:38:17.385670 13490 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33265 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:38:17.385784 13490 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:17.385931 13490 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:17.386732 13490 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:17.386883 13490 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
W20260924 04:38:17.387590 13490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:38:17.506024 13493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:17.506372 13493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:17.506438 13493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:17.512617 13493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:17.512702 13493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:17.512804 13493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:38:17.520646 13493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43891
--tserver_master_addrs=127.0.98.62:33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:17.522292 13493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:17.523591 13493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:17.535715 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:17.535844 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:17.535723 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:17.536895 13493 server_base.cc:1034] running on GCE node
I20260924 04:38:17.537338 13493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:17.538003 13493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:17.539186 13493 hybrid_clock.cc:648] HybridClock initialized: now 1790224697539155 us; error 50 us; skew 500 ppm
Sep 24 04:38:17 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:17.542243 13493 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:38:17.543299 13493 webserver.cc:460] Webserver started at http://127.0.98.3:43891/ using document root <none> and password file <none>
I20260924 04:38:17.543891 13493 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:17.544001 13493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:38:17 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:17.565062 13493 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:38:17.567418 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:17.568744 13493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:38:17.568849 13493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "bf64e9b13ebe4b0bab643bf88bef6c13"
format_stamp: "Formatted at 2026-09-24 04:38:16 on dist-test-slave-jlnt"
server_key: "62679fa2d3739b8f8dd27b24d6cd4a72"
server_key_iv: "fba0d5c8b8bd038140d0a564d09786b9"
server_key_version: "kuduclusterkey@0"
I20260924 04:38:17.569473 13493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:17.591543 13493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:17.595130 13493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:17.595335 13493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:17.595968 13493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:17.596827 13493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:17.596872 13493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:17.596911 13493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:17.596966 13493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:17.608043 13493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33439
I20260924 04:38:17.611070 13623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33439 every 8 connection(s)
I20260924 04:38:17.611547 13493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:38:17.620852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 13493
Sep 24 04:38:17 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:17.629276 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:17.615717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49881 (local address 127.0.98.62:33265)
0924 04:38:17.615854 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:17.615858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:17.616992 (+ 1134us) server_negotiation.cc:407] Connection header received
0924 04:38:17.618245 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:17.618249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:17.618315 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:17.618368 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:17.619937 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.620777 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.622328 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.622561 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.622674 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:17.625372 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:17.625376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:17.625380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:17.625407 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:17.627108 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:17.627710 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:17.627714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:17.627716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:17.627802 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:17.628144 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:17.628148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:17.628149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:17.628326 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:17.628433 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:17.628933 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 04:38:17.629046 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:38:17.630313 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:17.616301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33265 (local address 127.0.98.3:49881)
0924 04:38:17.616830 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:17.616885 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:17.617991 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:17.618755 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:17.618765 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:17.619096 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:17.619759 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:17.619779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.620940 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.620943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:17.622181 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:17.622217 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.622714 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.622718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:17.622929 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:17.623566 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:17.623582 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:17.625197 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:17.627245 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:17.627251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:17.627267 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:17.627599 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:17.627893 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:17.627898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:17.627900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:17.628042 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:17.628430 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:17.628436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:17.628692 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 04:38:17.628878 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 04:38:17.629232 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":135,"threads_started":1}
I20260924 04:38:17.631616 13624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33265
I20260924 04:38:17.631863 13624 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:17.632421 13624 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:17.633337 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:17.621641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.0.98.62:33265)
0924 04:38:17.621929 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:17.621934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:17.622225 (+ 291us) server_negotiation.cc:407] Connection header received
0924 04:38:17.622799 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:17.622802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:17.622845 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:17.622929 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:17.625020 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.625714 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.626937 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:17.627319 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:17.627418 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:17.629010 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:17.629014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:17.629017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:17.629043 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:17.631261 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:17.631763 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:17.631766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:17.631768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:17.631851 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:17.632242 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:17.632245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:17.632247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:17.632442 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:17.632643 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:17.633008 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 04:38:17.633161 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":97,"threads_started":1}
W20260924 04:38:17.634028 13624 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33265 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:38:17.634183 13624 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:17.634375 13624 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:17.635321 13624 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:17.635497 13624 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
W20260924 04:38:17.636291 13624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:38:17 dist-test-slave-jlnt krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790224697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:18.103220 13356 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:18.103533 13356 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:18.104601 13157 ts_manager.cc:194] Registered new tserver with Master: 42f9fe2bcacb441a817157b3097a22d2 (127.0.98.1:39925)
I20260924 04:38:18.106335 13157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38415
Sep 24 04:38:18 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:18.387789 13490 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:18.388062 13490 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:18.388761 13157 ts_manager.cc:194] Registered new tserver with Master: ccdc79361663476ba6f0364130603e5e (127.0.98.2:44107)
I20260924 04:38:18.389506 13157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47223
Sep 24 04:38:18 dist-test-slave-jlnt krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:38:18.636529 13624 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:18.636861 13624 heartbeater.cc:511] Master 127.0.98.62:33265 requested a full tablet report, sending...
I20260924 04:38:18.637641 13157 ts_manager.cc:194] Registered new tserver with Master: bf64e9b13ebe4b0bab643bf88bef6c13 (127.0.98.3:33439)
I20260924 04:38:18.638394 13157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49881
I20260924 04:38:18.641974 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:38:18.669593 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.649666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.0.98.62:33265)
0924 04:38:18.657351 (+ 7685us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:18.657356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:18.657702 (+ 346us) server_negotiation.cc:407] Connection header received
0924 04:38:18.657759 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:18.657762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:18.657816 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:18.657909 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:18.660718 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.661611 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.663107 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.663492 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.663573 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.665506 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:18.665510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:18.665513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:18.665540 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:18.667516 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:18.668037 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:18.668041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:18.668042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:18.668135 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:18.668479 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:18.668482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:18.668484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:18.668705 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:18.668813 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:18.669244 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 04:38:18.669403 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7575,"thread_start_us":83,"threads_started":1}
I20260924 04:38:18.674556 13157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:38:18.679540 13157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:38:18.745695 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.725739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39925 (local address 127.0.0.1:57184)
0924 04:38:18.729286 (+ 3547us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:18.729300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:18.729389 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:18.738448 (+ 9059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:18.738454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:18.738478 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:18.738624 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.738631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.740079 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.740083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.741558 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.741570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.745172 (+ 3602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.745175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.745287 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.745323 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:38:18.745430 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:38:18.745541 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3482,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260924 04:38:18.748769 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.726475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.0.98.1:39925)
0924 04:38:18.737659 (+ 11184us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:18.737667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:18.737694 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:38:18.737805 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:18.737812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:18.738070 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:18.738237 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:18.738768 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.739944 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.744050 (+ 4106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.744847 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.744944 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.745093 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:18.748514 (+ 3421us) server_negotiation.cc:299] Negotiation successful
0924 04:38:18.748623 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11069,"thread_start_us":75,"threads_started":1}
I20260924 04:38:18.750366 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.726188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33439 (local address 127.0.0.1:38876)
0924 04:38:18.737298 (+ 11110us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:18.737314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:18.737409 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:18.742649 (+ 5240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:18.742652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:18.742668 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:18.742747 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.742753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.746310 (+ 3557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.746313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.747404 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.747414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.750005 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.750009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.750099 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.750124 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:38:18.750172 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:38:18.750236 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11039,"thread_start_us":86,"threads_started":1}
I20260924 04:38:18.750737 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.724530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44107 (local address 127.0.0.1:56828)
0924 04:38:18.727684 (+ 3154us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:18.727740 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:18.727869 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:18.740923 (+ 13054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:18.740926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:18.740944 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:18.741049 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.741056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.743541 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.743545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.744713 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.744724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.750389 (+ 5665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.750393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.750479 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.750500 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:38:18.750551 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:38:18.750615 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2971,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260924 04:38:18.751304 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.726897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.0.98.3:33439)
0924 04:38:18.742031 (+ 15134us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:18.742038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:18.742064 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:38:18.742138 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:18.742143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:18.742340 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:18.742482 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:18.742887 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.743927 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.747553 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.748370 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.750898 (+ 2528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.751046 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:18.751093 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 04:38:18.751179 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14837,"thread_start_us":86,"threads_started":1}
I20260924 04:38:18.751564 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.724655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.0.98.2:44107)
0924 04:38:18.740433 (+ 15778us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:18.740440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:18.740467 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:38:18.740544 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:18.740549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:18.740758 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:18.742038 (+ 1280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:18.742214 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.743401 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.748775 (+ 5374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.749666 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.751137 (+ 1471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.751276 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:18.751327 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 04:38:18.751433 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15650,"thread_start_us":110,"threads_started":1}
I20260924 04:38:18.752305 13290 tablet_service.cc:1467] Processing CreateTablet for tablet d2a110d0162b4b4d9257d2aa572c8a23 (DEFAULT_TABLE table=test-table [id=7e81252e8d91414c8cb6ceb021921c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:18.753454 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a110d0162b4b4d9257d2aa572c8a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:18.754369 13424 tablet_service.cc:1467] Processing CreateTablet for tablet d2a110d0162b4b4d9257d2aa572c8a23 (DEFAULT_TABLE table=test-table [id=7e81252e8d91414c8cb6ceb021921c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:18.754403 13539 tablet_service.cc:1467] Processing CreateTablet for tablet d2a110d0162b4b4d9257d2aa572c8a23 (DEFAULT_TABLE table=test-table [id=7e81252e8d91414c8cb6ceb021921c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:18.755318 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a110d0162b4b4d9257d2aa572c8a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:18.755373 13539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a110d0162b4b4d9257d2aa572c8a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:18.765529 13661 tablet_bootstrap.cc:492] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: Bootstrap starting.
I20260924 04:38:18.767910 13661 tablet_bootstrap.cc:654] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:18.768806 13661 log.cc:824] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:18.769574 13660 tablet_bootstrap.cc:492] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2: Bootstrap starting.
I20260924 04:38:18.771893 13660 tablet_bootstrap.cc:654] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:18.772751 13660 log.cc:824] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:18.777534 13662 tablet_bootstrap.cc:492] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: Bootstrap starting.
I20260924 04:38:18.779742 13662 tablet_bootstrap.cc:654] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:18.780606 13662 log.cc:824] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:18.783044 13661 tablet_bootstrap.cc:492] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: No bootstrap required, opened a new log
I20260924 04:38:18.783245 13661 ts_tablet_manager.cc:1397] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260924 04:38:18.787837 13661 raft_consensus.cc:348] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } }
I20260924 04:38:18.788280 13661 raft_consensus.cc:374] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:18.788389 13661 raft_consensus.cc:729] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdc79361663476ba6f0364130603e5e, State: Initialized, Role: FOLLOWER
I20260924 04:38:18.788993 13661 consensus_queue.cc:260] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } }
I20260924 04:38:18.790517 13661 ts_tablet_manager.cc:1428] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:38:18.790922 13490 heartbeater.cc:503] Master 127.0.98.62:33265 was elected leader, sending a full tablet report...
I20260924 04:38:18.797032 13660 tablet_bootstrap.cc:492] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2: No bootstrap required, opened a new log
I20260924 04:38:18.797343 13660 ts_tablet_manager.cc:1397] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.002s
I20260924 04:38:18.803105 13660 raft_consensus.cc:348] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } }
I20260924 04:38:18.803470 13660 raft_consensus.cc:374] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:18.803575 13660 raft_consensus.cc:729] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f9fe2bcacb441a817157b3097a22d2, State: Initialized, Role: FOLLOWER
I20260924 04:38:18.804150 13660 consensus_queue.cc:260] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } }
I20260924 04:38:18.806691 13662 tablet_bootstrap.cc:492] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: No bootstrap required, opened a new log
I20260924 04:38:18.806962 13662 ts_tablet_manager.cc:1397] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20260924 04:38:18.811376 13662 raft_consensus.cc:348] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } }
I20260924 04:38:18.812431 13660 ts_tablet_manager.cc:1428] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2: Time spent starting tablet: real 0.015s user 0.001s sys 0.005s
I20260924 04:38:18.812844 13356 heartbeater.cc:503] Master 127.0.98.62:33265 was elected leader, sending a full tablet report...
I20260924 04:38:18.817719 13662 raft_consensus.cc:374] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:18.817864 13662 raft_consensus.cc:729] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf64e9b13ebe4b0bab643bf88bef6c13, State: Initialized, Role: FOLLOWER
I20260924 04:38:18.818413 13662 consensus_queue.cc:260] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } }
I20260924 04:38:18.819536 13662 ts_tablet_manager.cc:1428] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260924 04:38:18.819912 13624 heartbeater.cc:503] Master 127.0.98.62:33265 was elected leader, sending a full tablet report...
W20260924 04:38:18.821704 13357 tablet.cc:2369] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:38:18.860446 13491 tablet.cc:2369] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:38:18.866487 13625 tablet.cc:2369] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:38:18.929376 13666 raft_consensus.cc:482] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:18.929626 13666 raft_consensus.cc:504] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } }
I20260924 04:38:18.931044 13666 leader_election.cc:288] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42f9fe2bcacb441a817157b3097a22d2 (127.0.98.1:39925), bf64e9b13ebe4b0bab643bf88bef6c13 (127.0.98.3:33439)
I20260924 04:38:18.936834 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.931747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40745 (local address 127.0.98.1:39925)
0924 04:38:18.931867 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:18.931872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:18.932048 (+ 176us) server_negotiation.cc:407] Connection header received
0924 04:38:18.932259 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:18.932262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:18.932312 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:18.932357 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:18.932791 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.933614 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.935463 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.936113 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.936205 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.936265 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:18.936627 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 04:38:18.936711 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:38:18.936896 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.931760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54871 (local address 127.0.98.3:33439)
0924 04:38:18.931880 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:18.931886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:18.932150 (+ 264us) server_negotiation.cc:407] Connection header received
0924 04:38:18.932335 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:18.932338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:18.932398 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:18.932443 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:18.932863 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.933647 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.935323 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.935949 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.936025 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.936089 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:18.936703 (+ 614us) server_negotiation.cc:299] Negotiation successful
0924 04:38:18.936777 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:38:18.937213 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.931295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39925 (local address 127.0.98.2:40745)
0924 04:38:18.931950 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:18.931964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:18.932133 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:18.932487 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:18.932489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:18.932504 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:18.932658 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.932664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.933765 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.933768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.935275 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.935285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.936379 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.936382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.936479 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.936502 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:38:18.936965 (+ 463us) client_negotiation.cc:240] Negotiation successful
0924 04:38:18.937032 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":76,"thread_start_us":150,"threads_started":1}
I20260924 04:38:18.937713 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.931366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33439 (local address 127.0.98.2:54871)
0924 04:38:18.932062 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:18.932074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:18.932219 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:18.932569 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:18.932571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:18.932585 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:18.932741 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.932747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.933785 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.933788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.935185 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:18.935195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.936219 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.936222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:18.936563 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.936587 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:38:18.937466 (+ 879us) client_negotiation.cc:240] Negotiation successful
0924 04:38:18.937534 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":66,"thread_start_us":102,"threads_started":1}
I20260924 04:38:18.937894 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a110d0162b4b4d9257d2aa572c8a23" candidate_uuid: "ccdc79361663476ba6f0364130603e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f9fe2bcacb441a817157b3097a22d2" is_pre_election: true
I20260924 04:38:18.938205 13310 raft_consensus.cc:2454] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccdc79361663476ba6f0364130603e5e in term 0.
I20260924 04:38:18.938300 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a110d0162b4b4d9257d2aa572c8a23" candidate_uuid: "ccdc79361663476ba6f0364130603e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" is_pre_election: true
I20260924 04:38:18.938565 13578 raft_consensus.cc:2454] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccdc79361663476ba6f0364130603e5e in term 0.
I20260924 04:38:18.938726 13379 leader_election.cc:302] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f9fe2bcacb441a817157b3097a22d2, ccdc79361663476ba6f0364130603e5e; no voters:
I20260924 04:38:18.939011 13666 raft_consensus.cc:2781] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:38:18.939107 13666 raft_consensus.cc:482] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:38:18.939157 13666 raft_consensus.cc:3037] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:18.940425 13666 raft_consensus.cc:504] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } }
I20260924 04:38:18.940765 13666 leader_election.cc:288] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [CANDIDATE]: Term 1 election: Requested vote from peers 42f9fe2bcacb441a817157b3097a22d2 (127.0.98.1:39925), bf64e9b13ebe4b0bab643bf88bef6c13 (127.0.98.3:33439)
I20260924 04:38:18.941108 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a110d0162b4b4d9257d2aa572c8a23" candidate_uuid: "ccdc79361663476ba6f0364130603e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f9fe2bcacb441a817157b3097a22d2"
I20260924 04:38:18.941169 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a110d0162b4b4d9257d2aa572c8a23" candidate_uuid: "ccdc79361663476ba6f0364130603e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13"
I20260924 04:38:18.941242 13310 raft_consensus.cc:3037] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:18.941298 13578 raft_consensus.cc:3037] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:18.942454 13310 raft_consensus.cc:2454] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccdc79361663476ba6f0364130603e5e in term 1.
I20260924 04:38:18.942478 13578 raft_consensus.cc:2454] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccdc79361663476ba6f0364130603e5e in term 1.
I20260924 04:38:18.942763 13378 leader_election.cc:302] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf64e9b13ebe4b0bab643bf88bef6c13, ccdc79361663476ba6f0364130603e5e; no voters:
I20260924 04:38:18.942982 13666 raft_consensus.cc:2781] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:38:18.943199 13666 raft_consensus.cc:686] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 1 LEADER]: Becoming Leader. State: Replica: ccdc79361663476ba6f0364130603e5e, State: Running, Role: LEADER
I20260924 04:38:18.943523 13666 consensus_queue.cc:237] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } }
I20260924 04:38:18.946983 13158 catalog_manager.cc:5495] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e reported cstate change: term changed from 0 to 1, leader changed from <none> to ccdc79361663476ba6f0364130603e5e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ccdc79361663476ba6f0364130603e5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccdc79361663476ba6f0364130603e5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 } health_report { overall_health: UNKNOWN } } }
I20260924 04:38:18.959964 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:18.956525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.0.98.2:44107)
0924 04:38:18.956647 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:18.956652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:18.956663 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:38:18.956732 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:18.956734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:18.956784 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:18.956894 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:38:18.957170 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.957692 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.958542 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:18.958768 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:18.958874 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:18.959191 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:38:18.959585 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:38:18.959715 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:18.959756 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 04:38:18.959806 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:38:18.965960 13578 raft_consensus.cc:1261] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [term 1 FOLLOWER]: Refusing update from remote peer ccdc79361663476ba6f0364130603e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:18.965960 13310 raft_consensus.cc:1261] T d2a110d0162b4b4d9257d2aa572c8a23 P 42f9fe2bcacb441a817157b3097a22d2 [term 1 FOLLOWER]: Refusing update from remote peer ccdc79361663476ba6f0364130603e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:18.966657 13666 consensus_queue.cc:1035] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf64e9b13ebe4b0bab643bf88bef6c13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:18.966878 13678 consensus_queue.cc:1035] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f9fe2bcacb441a817157b3097a22d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:18.978124 13681 mvcc.cc:205] Tried to move back new op lower bound from 7332760366957699072 to 7332760366875365376. Current Snapshot: MvccSnapshot[applied={T|T < 7332760366957699072}]
I20260924 04:38:18.980214 13682 mvcc.cc:205] Tried to move back new op lower bound from 7332760366957699072 to 7332760366875365376. Current Snapshot: MvccSnapshot[applied={T|T < 7332760366957699072}]
I20260924 04:38:18.980448 13683 mvcc.cc:205] Tried to move back new op lower bound from 7332760366957699072 to 7332760366875365376. Current Snapshot: MvccSnapshot[applied={T|T < 7332760366957699072}]
I20260924 04:38:18.988878 13158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:38:18.989063 13158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:38:18.991700 13158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccbf8d08002449c1848c5de76bfae585: Sending DeleteTablet for 3 replicas of tablet d2a110d0162b4b4d9257d2aa572c8a23
I20260924 04:38:18.993070 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a110d0162b4b4d9257d2aa572c8a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56828
I20260924 04:38:18.993562 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a110d0162b4b4d9257d2aa572c8a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57184
I20260924 04:38:18.994000 13539 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a110d0162b4b4d9257d2aa572c8a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38876
I20260924 04:38:18.995594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 13196
I20260924 04:38:18.999540 13691 tablet_replica.cc:330] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: stopping tablet replica
I20260924 04:38:18.999876 13691 raft_consensus.cc:2229] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:38:19.000293 13691 raft_consensus.cc:2258] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:19.001844 13693 tablet_replica.cc:330] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: stopping tablet replica
I20260924 04:38:19.002465 13693 raft_consensus.cc:2229] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:38:19.002856 13693 raft_consensus.cc:2258] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:19.003787 13691 ts_tablet_manager.cc:1905] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:38:19.007537 13693 ts_tablet_manager.cc:1905] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:38:19.010193 13693 ts_tablet_manager.cc:1918] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:38:19.010298 13693 log.cc:1196] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/d2a110d0162b4b4d9257d2aa572c8a23
I20260924 04:38:19.010645 13693 ts_tablet_manager.cc:1939] T d2a110d0162b4b4d9257d2aa572c8a23 P bf64e9b13ebe4b0bab643bf88bef6c13: Deleting consensus metadata
I20260924 04:38:19.010825 13691 ts_tablet_manager.cc:1918] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:38:19.012004 13143 catalog_manager.cc:4841] TS bf64e9b13ebe4b0bab643bf88bef6c13 (127.0.98.3:33439): tablet d2a110d0162b4b4d9257d2aa572c8a23 (table test-table [id=7e81252e8d91414c8cb6ceb021921c9a]) successfully deleted
I20260924 04:38:19.012202 13691 log.cc:1196] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/d2a110d0162b4b4d9257d2aa572c8a23
I20260924 04:38:19.012595 13691 ts_tablet_manager.cc:1939] T d2a110d0162b4b4d9257d2aa572c8a23 P ccdc79361663476ba6f0364130603e5e: Deleting consensus metadata
W20260924 04:38:19.013376 13144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39925 (error 108)
I20260924 04:38:19.014256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 13360
I20260924 04:38:19.016389 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:19.016114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39925 (local address 127.0.0.1:57186)
0924 04:38:19.016241 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:19.016309 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:38:19.016570 13144 catalog_manager.cc:4584] TS 42f9fe2bcacb441a817157b3097a22d2 (127.0.98.1:39925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2a110d0162b4b4d9257d2aa572c8a23: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39925: connect: Connection refused (error 111)
I20260924 04:38:19.021528 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:19.021199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44107 (local address 127.0.0.1:56844)
0924 04:38:19.021398 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:19.021449 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 04:38:19.021626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 13493
I20260924 04:38:19.030263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 13125
I20260924 04:38:19.037478 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:38:19.381570 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:38:19.381667 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:38:19.725733 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:38:19.725 UTC [12192] LOG: received smart shutdown request
2026-09-24 04:38:19.730 UTC [12192] LOG: background worker "logical replication launcher" (PID 12201) exited with exit code 1
2026-09-24 04:38:19.730 UTC [12196] LOG: shutting down
2026-09-24 04:38:19.898 UTC [12192] LOG: database system is shut down
2026-09-24T04:38:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46840 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:38:20 dist-test-slave-jlnt krb5kdc[13703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:38:20 dist-test-slave-jlnt krb5kdc[13703](info): set up 2 sockets
Sep 24 04:38:20 dist-test-slave-jlnt krb5kdc[13703](info): commencing operation
krb5kdc: starting...
W20260924 04:38:21.057035 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:38:21.172456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35333
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35333,127.0.98.61:37721,127.0.98.60:37293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:21.280164 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:21.280395 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:21.280433 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:21.284268 13716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:21.284315 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:21.284335 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:21.284351 13716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:21.284366 13716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:21.289353 13716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35333,127.0.98.61:37721,127.0.98.60:37293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:21.290246 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:21.291127 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:21.299088 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:21.299211 13716 server_base.cc:1034] running on GCE node
W20260924 04:38:21.299105 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:21.299108 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:21.299894 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:21.300897 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:21.302093 13716 hybrid_clock.cc:648] HybridClock initialized: now 1790224701302059 us; error 44 us; skew 500 ppm
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:21.304702 13716 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:38:21.305689 13716 webserver.cc:460] Webserver started at http://127.0.98.62:43729/ using document root <none> and password file <none>
I20260924 04:38:21.306406 13716 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:21.306495 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:21.306710 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:21.308612 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "a3163c3cecea48d7be519d89cc3c1c22"
format_stamp: "Formatted at 2026-09-24 04:38:21 on dist-test-slave-jlnt"
server_key: "3790a217416c6a5e7fd9bf2a48becd88"
server_key_iv: "96ed77c6a704d32a63482126d0d09007"
server_key_version: "encryptionkey@0"
I20260924 04:38:21.309185 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3163c3cecea48d7be519d89cc3c1c22"
format_stamp: "Formatted at 2026-09-24 04:38:21 on dist-test-slave-jlnt"
server_key: "3790a217416c6a5e7fd9bf2a48becd88"
server_key_iv: "96ed77c6a704d32a63482126d0d09007"
server_key_version: "encryptionkey@0"
I20260924 04:38:21.312577 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:38:21.314992 13731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:21.316143 13716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:21.316272 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "a3163c3cecea48d7be519d89cc3c1c22"
format_stamp: "Formatted at 2026-09-24 04:38:21 on dist-test-slave-jlnt"
server_key: "3790a217416c6a5e7fd9bf2a48becd88"
server_key_iv: "96ed77c6a704d32a63482126d0d09007"
server_key_version: "encryptionkey@0"
I20260924 04:38:21.316380 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:21.349462 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:21.352366 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:21.352557 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:21.359557 13716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35333
I20260924 04:38:21.359593 13783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35333 every 8 connection(s)
I20260924 04:38:21.360453 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:38:21.363106 13784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:21.366582 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:38:21.368355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 13716
I20260924 04:38:21.368481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:38:21.368736 392 external_mini_cluster.cc:1428] Setting key 1dba883d6b46407455f395006294e7a2
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 04:38:21.382454 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.368754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35333 (local address 127.0.0.1:52056)
0924 04:38:21.369286 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.369337 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:21.370553 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:21.371303 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:21.371307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:21.371731 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:21.372131 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.372152 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.373310 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.373313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.374633 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.374647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.375279 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.375287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.375429 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.376025 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:21.376049 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:21.377696 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:21.379846 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.379853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.379860 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.380074 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.380399 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.380403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.380405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.380506 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.381256 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:21.381260 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:21.381378 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 04:38:21.381513 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 04:38:21.381736 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":137,"threads_started":1}
I20260924 04:38:21.382459 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.368735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.0.98.62:35333)
0924 04:38:21.369281 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.369297 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.369454 (+ 157us) server_negotiation.cc:407] Connection header received
0924 04:38:21.370859 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.370868 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.371099 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.371318 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.372303 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.373130 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.374772 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.375124 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.375251 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.377834 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.377839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.377848 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.377867 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.379698 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.380191 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.380196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.380198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.380299 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.380600 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.380606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.380608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.381082 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.381263 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.381626 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.381882 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":424,"spinlock_wait_cycles":49024,"thread_start_us":148,"threads_started":1}
I20260924 04:38:21.384776 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:38:21.385489 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.385250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37721 (local address 127.0.0.1:55916)
0924 04:38:21.385370 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.385440 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 04:38:21.385767 13734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
I20260924 04:38:21.386543 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.386347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37721 (local address 127.0.0.1:55920)
0924 04:38:21.386449 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.386494 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:38:21.386672 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37721: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:38:21.393929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37721
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35333,127.0.98.61:37721,127.0.98.60:37293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:38:21.435909 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } attempt: 1
I20260924 04:38:21.436764 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.436478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37721 (local address 127.0.0.1:55930)
0924 04:38:21.436629 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.436690 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 04:38:21.437501 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.437280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37721 (local address 127.0.0.1:55944)
0924 04:38:21.437400 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.437449 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 04:38:21.437628 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37721: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
W20260924 04:38:21.501248 13791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:21.501505 13791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:21.501592 13791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:21.505511 13791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:21.505578 13791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:21.505614 13791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:21.505668 13791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:21.505705 13791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:21.505793 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } attempt: 2
I20260924 04:38:21.506546 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.506263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37721 (local address 127.0.0.1:55950)
0924 04:38:21.506408 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.506473 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 04:38:21.507274 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.507065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37721 (local address 127.0.0.1:55960)
0924 04:38:21.507170 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.507216 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:38:21.507478 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37721: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37721: connect: Connection refused (error 111)
I20260924 04:38:21.511356 13791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35333,127.0.98.61:37721,127.0.98.60:37293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:21.512336 13791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:21.513307 13791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:21.521436 13796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:21.521502 13791 server_base.cc:1034] running on GCE node
W20260924 04:38:21.521432 13799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:21.521436 13797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:21.522173 13791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:21.523167 13791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:21.524364 13791 hybrid_clock.cc:648] HybridClock initialized: now 1790224701524337 us; error 48 us; skew 500 ppm
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:21.527060 13791 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:38:21.527874 13791 webserver.cc:460] Webserver started at http://127.0.98.61:44821/ using document root <none> and password file <none>
I20260924 04:38:21.528414 13791 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:21.528504 13791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:21.528725 13791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:21.530725 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/data/instance:
uuid: "c59fb9b60e244e4f879334cab3670ac1"
format_stamp: "Formatted at 2026-09-24 04:38:21 on dist-test-slave-jlnt"
server_key: "6eca8d4f46e2c71b260d0d4b536a282b"
server_key_iv: "3460f5ec409b382f8c0835ffb98006b3"
server_key_version: "encryptionkey@0"
I20260924 04:38:21.531272 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/wal/instance:
uuid: "c59fb9b60e244e4f879334cab3670ac1"
format_stamp: "Formatted at 2026-09-24 04:38:21 on dist-test-slave-jlnt"
server_key: "6eca8d4f46e2c71b260d0d4b536a282b"
server_key_iv: "3460f5ec409b382f8c0835ffb98006b3"
server_key_version: "encryptionkey@0"
I20260924 04:38:21.534752 13791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:38:21.537037 13806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:21.538287 13791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:21.538437 13791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/wal
uuid: "c59fb9b60e244e4f879334cab3670ac1"
format_stamp: "Formatted at 2026-09-24 04:38:21 on dist-test-slave-jlnt"
server_key: "6eca8d4f46e2c71b260d0d4b536a282b"
server_key_iv: "3460f5ec409b382f8c0835ffb98006b3"
server_key_version: "encryptionkey@0"
I20260924 04:38:21.538544 13791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:21.559295 13791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:21.561947 13791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:21.562137 13791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:21.568603 13791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37721
I20260924 04:38:21.568661 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37721 every 8 connection(s)
I20260924 04:38:21.569550 13791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:38:21.572093 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:21.575719 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:38:21.579236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 13791
I20260924 04:38:21.579357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-1/wal/instance
I20260924 04:38:21.579658 392 external_mini_cluster.cc:1428] Setting key 44e0a7656cc8ed310c27276179400201
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 04:38:21.589725 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.577560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.0.98.62:35333)
0924 04:38:21.577701 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.577705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.578476 (+ 771us) server_negotiation.cc:407] Connection header received
0924 04:38:21.579420 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.579424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.579484 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.579573 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.580689 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.581448 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.582975 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.583208 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.583299 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.586251 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.586254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.586256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.586278 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.587812 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.588305 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.588307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.588308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.588387 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.588688 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.588690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.588692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.588849 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.588924 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.589484 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.589579 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 04:38:21.590595 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.577894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35333 (local address 127.0.0.1:52066)
0924 04:38:21.578335 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.578375 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:21.579168 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:21.579756 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:21.579768 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:21.580082 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:21.580532 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.580544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.581576 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.581580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.582839 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.582854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.583370 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.583377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.583601 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.584217 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:21.584257 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:21.586079 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:21.587923 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.587932 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.587948 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.588211 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.588485 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.588487 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.588488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.588610 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.588950 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:21.588958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:21.589270 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 04:38:21.589425 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:38:21.589703 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260924 04:38:21.591907 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:38:21.598628 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } attempt: 3
I20260924 04:38:21.601043 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.592361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37721 (local address 127.0.0.1:55970)
0924 04:38:21.592468 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.592483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:21.592552 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:21.593106 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:21.593109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:21.593306 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:21.593373 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.593378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.594315 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.594318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.594848 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.594855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.595409 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.595412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.595500 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.595954 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:21.595970 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:21.597342 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:21.599531 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.599537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.599540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.599763 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.600044 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.600049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.600051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.600144 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.600584 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:21.600588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:21.600681 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 04:38:21.600771 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:38:21.600862 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:38:21.601502 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.592512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.0.98.61:37721)
0924 04:38:21.592802 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.592810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.592826 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:38:21.592855 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.592859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.592989 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.593130 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.593474 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.594192 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.594956 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.595304 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.595413 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.597484 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.597493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.597509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.597528 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.599436 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.599866 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.599870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.599871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.599949 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.600233 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.600238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.600239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.600478 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.600601 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.600874 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.601010 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":132,"threads_started":1}
I20260924 04:38:21.603217 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:38:21.603892 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.603656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52546)
0924 04:38:21.603759 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.603840 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:38:21.604239 13807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:38:21.605123 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.604880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52558)
0924 04:38:21.604990 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.605043 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:38:21.605362 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:38:21.608186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37293
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35333,127.0.98.61:37721,127.0.98.60:37293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:38:21.608779 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.599033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37721 (local address 127.0.0.1:55974)
0924 04:38:21.599152 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.599172 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:21.599234 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:21.599757 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:21.599760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:21.599930 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:21.600503 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.600512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.601549 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.601571 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.602234 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.602248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.602791 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.602795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.602895 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.603560 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:21.603571 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:21.605009 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:21.607145 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.607147 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.607149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.607393 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.607685 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.607688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.607690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.607788 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.608142 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:21.608145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:21.608257 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 04:38:21.608376 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:38:21.608526 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 04:38:21.609419 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.599130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.0.98.61:37721)
0924 04:38:21.599485 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.599494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.599509 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:38:21.599549 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.599554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.599607 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.599707 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.600637 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.601424 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.602340 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.602654 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.602756 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.605230 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.605237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.605243 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.605268 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.607025 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.607487 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.607493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.607498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.607584 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.607889 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.607891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.607892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.608045 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.608117 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.609009 (+ 892us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.609166 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
I20260924 04:38:21.610314 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:38:21.611017 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.610718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52572)
0924 04:38:21.610869 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.610931 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 04:38:21.611785 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.611477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52578)
0924 04:38:21.611639 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.611698 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 04:38:21.611922 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
I20260924 04:38:21.643100 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } attempt: 1
I20260924 04:38:21.643926 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.643595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52588)
0924 04:38:21.643754 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.643833 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 04:38:21.644641 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.644421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52590)
0924 04:38:21.644533 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.644584 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:38:21.644783 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
I20260924 04:38:21.654567 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } attempt: 1
I20260924 04:38:21.655247 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.655009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52602)
0924 04:38:21.655132 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.655184 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:38:21.655949 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.655734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52616)
0924 04:38:21.655851 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.655901 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:38:21.656071 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
W20260924 04:38:21.717296 13867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:21.717579 13867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:21.717664 13867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:38:21.719949 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } attempt: 2
I20260924 04:38:21.720701 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.720399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52624)
0924 04:38:21.720548 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.720615 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:38:21.721429 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.721225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52634)
0924 04:38:21.721323 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.721370 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:38:21.721568 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
W20260924 04:38:21.722137 13867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:21.722179 13867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:21.722219 13867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:21.722254 13867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:21.722308 13867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:21.724226 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } attempt: 2
I20260924 04:38:21.724841 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.724596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52638)
0924 04:38:21.724713 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.724765 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 04:38:21.725567 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.725371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52654)
0924 04:38:21.725479 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.725522 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 04:38:21.725737 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37293: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37293: connect: Connection refused (error 111)
I20260924 04:38:21.727809 13867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35333,127.0.98.61:37721,127.0.98.60:37293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:21.728731 13867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:21.729743 13867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:21.737423 13872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:21.737423 13875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:21.737480 13873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:21.737838 13867 server_base.cc:1034] running on GCE node
I20260924 04:38:21.738395 13867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:21.739373 13867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:21.740547 13867 hybrid_clock.cc:648] HybridClock initialized: now 1790224701740532 us; error 23 us; skew 500 ppm
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:21.743407 13867 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:38:21.744244 13867 webserver.cc:460] Webserver started at http://127.0.98.60:43797/ using document root <none> and password file <none>
I20260924 04:38:21.744820 13867 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:21.744913 13867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:21.745131 13867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:21.747062 13867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/data/instance:
uuid: "3aa7a7c8ba334fc2ad678854a306ecb8"
format_stamp: "Formatted at 2026-09-24 04:38:21 on dist-test-slave-jlnt"
server_key: "594334f4047d99134f905fff79634bf8"
server_key_iv: "f20662d7df8a61c13d5ad581a6a264c8"
server_key_version: "encryptionkey@0"
I20260924 04:38:21.747609 13867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/wal/instance:
uuid: "3aa7a7c8ba334fc2ad678854a306ecb8"
format_stamp: "Formatted at 2026-09-24 04:38:21 on dist-test-slave-jlnt"
server_key: "594334f4047d99134f905fff79634bf8"
server_key_iv: "f20662d7df8a61c13d5ad581a6a264c8"
server_key_version: "encryptionkey@0"
I20260924 04:38:21.751055 13867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:38:21.753299 13882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:21.754472 13867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:21.754609 13867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/wal
uuid: "3aa7a7c8ba334fc2ad678854a306ecb8"
format_stamp: "Formatted at 2026-09-24 04:38:21 on dist-test-slave-jlnt"
server_key: "594334f4047d99134f905fff79634bf8"
server_key_iv: "f20662d7df8a61c13d5ad581a6a264c8"
server_key_version: "encryptionkey@0"
I20260924 04:38:21.754712 13867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:21.772519 13867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:21.775632 13867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:21.775817 13867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:21.782449 13867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37293
I20260924 04:38:21.782473 13934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37293 every 8 connection(s)
I20260924 04:38:21.783313 13867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:38:21.784461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 13867
I20260924 04:38:21.784559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790224545298817-392-0/minicluster-data/master-2/wal/instance
I20260924 04:38:21.784847 392 external_mini_cluster.cc:1428] Setting key 73691ede2e57b33965ba75d5534961d2
I20260924 04:38:21.786128 13935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:21.789659 13935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:21.798975 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.786117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.0.98.60:37293)
0924 04:38:21.786516 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.786530 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.786562 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:38:21.787173 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.787188 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.787487 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.787807 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.788567 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.789604 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.790524 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.790974 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.791141 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.793876 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.793882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.793893 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.793922 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.795940 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.796362 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.796368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.796373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.796481 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.796787 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.796790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.796791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.797236 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.797399 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.797729 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.798010 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260924 04:38:21.800901 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.791216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.0.98.62:35333)
0924 04:38:21.791355 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.791359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.791712 (+ 353us) server_negotiation.cc:407] Connection header received
0924 04:38:21.791881 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.791885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.791939 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.792009 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.792850 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.793432 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.794597 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.794920 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.795016 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.797586 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.797592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.797595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.797620 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.799141 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.799585 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.799588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.799590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.799675 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.800039 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.800045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.800051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.800208 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.800284 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.800652 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.800760 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:38:21.801111 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.791174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35333 (local address 127.0.0.1:52068)
0924 04:38:21.791574 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.791628 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:21.791755 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:21.792087 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:21.792093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:21.792545 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:21.792725 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.792736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.793558 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.793560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.794460 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.794474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.795049 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.795053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.795177 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.795651 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:21.795669 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:21.797419 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:21.799249 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.799258 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.799263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.799483 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.799805 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.799807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.799809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.799930 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.800309 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:21.800314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:21.800420 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:38:21.800558 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 04:38:21.800722 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":150,"threads_started":1}
I20260924 04:38:21.802136 13935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:38:21.809983 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.801325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.0.98.62:35333)
0924 04:38:21.801562 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.801566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.801577 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:38:21.801607 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.801611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.801649 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.801746 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.802430 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.802993 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.803959 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.804154 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.804252 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.806789 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.806793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.806795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.806818 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.808397 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.808794 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.808798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.808800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.808881 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.809191 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.809194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.809196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.809355 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.809572 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.809702 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.809807 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260924 04:38:21.810338 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.802565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37721 (local address 127.0.0.1:55982)
0924 04:38:21.802816 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.802830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:21.802910 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:21.803142 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:21.803144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:21.803263 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:21.803352 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.803356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.804298 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.804302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.804963 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.804972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.805672 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.805678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.805783 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.806151 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:21.806166 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:21.807343 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:21.808907 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.808912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.808914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.809100 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.809420 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.809424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.809426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.809509 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.809874 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:21.809877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:21.809959 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 04:38:21.810072 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 04:38:21.810202 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260924 04:38:21.810688 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.802682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.0.98.61:37721)
0924 04:38:21.802808 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.802813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.802873 (+ 60us) server_negotiation.cc:407] Connection header received
0924 04:38:21.802986 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.802989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.803031 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.803111 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.803478 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.804158 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.805086 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.805538 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.805664 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.807497 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.807500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.807502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.807525 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.808799 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.809240 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.809250 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.809252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.809327 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.809597 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.809601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.809603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.809775 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.809883 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.810250 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.810372 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:38:21.811426 13935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:38:21.816915 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } attempt: 3
I20260924 04:38:21.820030 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.811055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.0.98.61:37721)
0924 04:38:21.811160 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.811164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.811182 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:38:21.811372 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.811375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.811417 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.811487 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.812415 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.812898 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.813682 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.813908 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.814013 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.816499 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.816502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.816523 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.816552 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.818434 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.818821 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.818828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.818831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.818919 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.819188 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.819192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.819194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.819373 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.819566 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.819797 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.819898 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:38:21.820791 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.811844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52662)
0924 04:38:21.811921 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.811935 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:21.812002 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:21.812176 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:21.812177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:21.812290 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:21.812348 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.812375 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.813043 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.813046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.813638 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.813646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.814683 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.814687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.814782 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.815224 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:21.815241 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:21.816797 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:21.818866 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.818869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.818870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.819093 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.819845 (+ 752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.819848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.819849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.819938 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.820339 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:21.820344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:21.820422 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 04:38:21.820534 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 04:38:21.820659 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":3}
I20260924 04:38:21.820852 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } attempt: 3
I20260924 04:38:21.822312 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.811930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.0.98.60:37293)
0924 04:38:21.812024 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.812028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.812040 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:38:21.812076 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.812078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.812107 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.812168 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.812474 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.812932 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.813748 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.814548 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.814684 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.816969 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.816972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.816975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.817001 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.818759 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.819619 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.819625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.819629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.819713 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.820032 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.820035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.820037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.820230 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.820328 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.820588 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.822153 (+ 1565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1470,"server-negotiator.queue_time_us":31}
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:38:21.825904 13935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8: Bootstrap starting.
Sep 24 04:38:21 dist-test-slave-jlnt krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:38:21.827059 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.817281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52664)
0924 04:38:21.817419 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.817434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:21.817509 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:21.818889 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:21.818893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:21.819064 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:21.819183 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.819190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.820199 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.820204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.820985 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.820995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.821564 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.821567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.821646 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.822340 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:21.822357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:21.823607 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:21.825566 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.825569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.825571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.825758 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.826074 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.826077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.826079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.826164 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.826549 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:21.826554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:21.826649 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 04:38:21.826765 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 04:38:21.826879 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:38:21.827601 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.817531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.0.98.60:37293)
0924 04:38:21.818668 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.818677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.818691 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:38:21.818724 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.818727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.818763 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.819290 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.819384 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.820071 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.821148 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.821454 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.821562 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.823732 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.823735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.823738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.823759 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.825450 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.825855 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.825860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.825862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.825943 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.826268 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.826271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.826273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.826458 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.826556 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.827068 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.827206 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":105,"threads_started":1}
I20260924 04:38:21.828559 13935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:21.829521 13935 log.cc:824] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:21.829962 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.821208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37293 (local address 127.0.0.1:52680)
0924 04:38:21.821547 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:21.821562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:21.821639 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:21.821894 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:21.821900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:21.822084 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:21.822178 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.822187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.823051 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.823054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.823787 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:21.823799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.824396 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.824400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:21.824478 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.825038 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:21.825052 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:21.826380 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:21.828302 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.828306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.828308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.828567 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.828882 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:21.828884 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:21.828886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:21.828975 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:21.829441 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:21.829444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:21.829533 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:38:21.829657 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 04:38:21.829788 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260924 04:38:21.830400 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.821329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.0.98.60:37293)
0924 04:38:21.821598 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.821606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.821620 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:38:21.821712 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.821716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.821752 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.821867 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.822300 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.822937 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.823906 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.824269 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.824389 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.826499 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.826502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.826505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.826529 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.828192 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.828698 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.828702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.828703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.828780 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.829094 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.829097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.829098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.829354 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.829448 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.830139 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.830236 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":103,"threads_started":1}
I20260924 04:38:21.830567 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1: Bootstrap starting.
I20260924 04:38:21.831267 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:38:21.831897 13935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8: No bootstrap required, opened a new log
I20260924 04:38:21.833076 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:21.833894 13859 log.cc:824] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:21.836529 13935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } }
I20260924 04:38:21.836787 13935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:21.836860 13935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa7a7c8ba334fc2ad678854a306ecb8, State: Initialized, Role: FOLLOWER
I20260924 04:38:21.836911 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22: Bootstrap starting.
I20260924 04:38:21.837370 13935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } }
I20260924 04:38:21.838627 13935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:21.839318 13784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:21.840166 13784 log.cc:824] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:21.842639 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22: No bootstrap required, opened a new log
I20260924 04:38:21.844159 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.833643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.0.98.60:37293)
0924 04:38:21.833733 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.833737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.834094 (+ 357us) server_negotiation.cc:407] Connection header received
0924 04:38:21.834253 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.834255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.834296 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.834345 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.835226 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.835994 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.837047 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.837455 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.837526 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.839831 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.839835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.839838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.839865 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.842300 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.842853 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.842857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.842859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.842940 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.843250 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.843254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.843255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.843453 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.843572 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.843908 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.844016 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:38:21.844193 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.833336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.0.98.61:37721)
0924 04:38:21.833448 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.833454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.834047 (+ 593us) server_negotiation.cc:407] Connection header received
0924 04:38:21.834230 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.834233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.834274 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.834326 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.835882 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.836587 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.837657 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.838306 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.838395 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.840306 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.840310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.840313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.840338 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.842254 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.842893 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.842898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.842900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.842979 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.843272 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.843275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.843277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.843486 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.843577 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.843945 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.844048 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:38:21.842144 13867 master_runner.cc:186] Error getting master registration for 127.0.98.62:35333: OK, instance_id { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" instance_seqno: 1790224701357447 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:38:21.845664 13867 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:38:21.847025 13784 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } }
I20260924 04:38:21.847195 13784 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:21.847254 13784 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3163c3cecea48d7be519d89cc3c1c22, State: Initialized, Role: FOLLOWER
I20260924 04:38:21.847625 13784 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } }
I20260924 04:38:21.848094 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.848244 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:38:21.848836 13784 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:21.850993 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1: No bootstrap required, opened a new log
I20260924 04:38:21.851997 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:21.833051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.0.98.62:35333)
0924 04:38:21.833255 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:21.833262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:21.834831 (+ 1569us) server_negotiation.cc:407] Connection header received
0924 04:38:21.835005 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:21.835009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:21.835053 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:21.835106 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:21.840149 (+ 5043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.840936 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.846465 (+ 5529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:21.846770 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:21.846877 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:21.848396 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:21.848400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:21.848402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:21.848427 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:21.850127 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.850632 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.850637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.850641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.850731 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:21.851060 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:21.851063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:21.851065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:21.851265 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:21.851397 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:21.851750 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 04:38:21.851863 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260924 04:38:21.852561 13963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:38:21.852669 13963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:38:21.853197 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.853354 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 04:38:21.854072 13737 tablet.cc:2369] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:38:21.855549 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } }
I20260924 04:38:21.855760 13859 raft_consensus.cc:374] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:21.855818 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59fb9b60e244e4f879334cab3670ac1, State: Initialized, Role: FOLLOWER
I20260924 04:38:21.856267 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } }
I20260924 04:38:21.856742 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.853737 13716 master_runner.cc:186] Error getting master registration for 127.0.98.61:37721: OK, instance_id { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" instance_seqno: 1790224701566824 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:38:21.856973 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:38:21.856997 13716 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:38:21.858616 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:21.861474 13964 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:21.861600 13964 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } }
I20260924 04:38:21.862488 13964 leader_election.cc:288] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c59fb9b60e244e4f879334cab3670ac1 (127.0.98.61:37721), 3aa7a7c8ba334fc2ad678854a306ecb8 (127.0.98.60:37293)
I20260924 04:38:21.862973 13909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3163c3cecea48d7be519d89cc3c1c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" is_pre_election: true
I20260924 04:38:21.863212 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3163c3cecea48d7be519d89cc3c1c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59fb9b60e244e4f879334cab3670ac1" is_pre_election: true
I20260924 04:38:21.863243 13909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3163c3cecea48d7be519d89cc3c1c22 in term 0.
I20260924 04:38:21.863461 13833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3163c3cecea48d7be519d89cc3c1c22 in term 0.
I20260924 04:38:21.863703 13732 leader_election.cc:302] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa7a7c8ba334fc2ad678854a306ecb8, a3163c3cecea48d7be519d89cc3c1c22; no voters:
I20260924 04:38:21.863940 13964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:38:21.864023 13964 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:38:21.864064 13964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:21.865320 13964 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } }
I20260924 04:38:21.865633 13964 leader_election.cc:288] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [CANDIDATE]: Term 1 election: Requested vote from peers c59fb9b60e244e4f879334cab3670ac1 (127.0.98.61:37721), 3aa7a7c8ba334fc2ad678854a306ecb8 (127.0.98.60:37293)
I20260924 04:38:21.865860 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3163c3cecea48d7be519d89cc3c1c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59fb9b60e244e4f879334cab3670ac1"
I20260924 04:38:21.865995 13833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:21.866103 13909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3163c3cecea48d7be519d89cc3c1c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8"
I20260924 04:38:21.866204 13909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [term 0 FOLLOWER]: Advancing to term 1
W20260924 04:38:21.866241 13976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:38:21.866300 13976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:38:21.867213 13833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3163c3cecea48d7be519d89cc3c1c22 in term 1.
I20260924 04:38:21.867429 13909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3163c3cecea48d7be519d89cc3c1c22 in term 1.
I20260924 04:38:21.867590 13734 leader_election.cc:302] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3163c3cecea48d7be519d89cc3c1c22, c59fb9b60e244e4f879334cab3670ac1; no voters:
I20260924 04:38:21.867771 13964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 04:38:21.867868 13988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:38:21.867942 13988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:38:21.867961 13964 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [term 1 LEADER]: Becoming Leader. State: Replica: a3163c3cecea48d7be519d89cc3c1c22, State: Running, Role: LEADER
I20260924 04:38:21.868243 13964 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } }
I20260924 04:38:21.868479 13791 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260924 04:38:21.868574 13791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:38:21.870569 13991 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3163c3cecea48d7be519d89cc3c1c22. Latest consensus state: current_term: 1 leader_uuid: "a3163c3cecea48d7be519d89cc3c1c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.870703 13991 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:21.871104 13993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:38:21.872030 13993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:38:21.880592 13833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [term 1 FOLLOWER]: Refusing update from remote peer a3163c3cecea48d7be519d89cc3c1c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:21.880630 13908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [term 1 FOLLOWER]: Refusing update from remote peer a3163c3cecea48d7be519d89cc3c1c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:21.881242 13964 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:21.881944 13964 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:21.885907 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3163c3cecea48d7be519d89cc3c1c22. Latest consensus state: current_term: 1 leader_uuid: "a3163c3cecea48d7be519d89cc3c1c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.886013 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:38:21.888849 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3163c3cecea48d7be519d89cc3c1c22. Latest consensus state: current_term: 1 leader_uuid: "a3163c3cecea48d7be519d89cc3c1c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.888957 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:38:21.893962 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3163c3cecea48d7be519d89cc3c1c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.894065 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:21.894157 13994 mvcc.cc:205] Tried to move back new op lower bound from 7332760378895241216 to 7332760378855030784. Current Snapshot: MvccSnapshot[applied={T|T < 7332760378895241216}]
I20260924 04:38:21.895165 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3163c3cecea48d7be519d89cc3c1c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.895264 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:38:21.897832 13991 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3163c3cecea48d7be519d89cc3c1c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.897933 13991 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:21.899017 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3163c3cecea48d7be519d89cc3c1c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3163c3cecea48d7be519d89cc3c1c22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35333 } } peers { permanent_uuid: "c59fb9b60e244e4f879334cab3670ac1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37721 } } peers { permanent_uuid: "3aa7a7c8ba334fc2ad678854a306ecb8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37293 } } }
I20260924 04:38:21.899134 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:38:21.899817 13997 mvcc.cc:205] Tried to move back new op lower bound from 7332760378895241216 to 7332760378855030784. Current Snapshot: MvccSnapshot[applied={T|T < 7332760378895241216}]
I20260924 04:38:21.900028 13995 mvcc.cc:205] Tried to move back new op lower bound from 7332760378895241216 to 7332760378855030784. Current Snapshot: MvccSnapshot[applied={T|T < 7332760378895241216}]
I20260924 04:38:21.905735 13993 catalog_manager.cc:1347] Generated new cluster ID: d2fa9de2a2dc4043a126a62590b6a43d
I20260924 04:38:21.905802 13993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:38:21.923820 13993 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:38:21.925073 13993 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:38:21.945945 13993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3163c3cecea48d7be519d89cc3c1c22: Generated new TSK 0
I20260924 04:38:21.946832 13993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:38:22.854122 13963 catalog_manager.cc:1259] Loaded cluster ID: d2fa9de2a2dc4043a126a62590b6a43d
I20260924 04:38:22.854212 13963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8: loading cluster ID for follower catalog manager: success
I20260924 04:38:22.855787 13963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8: acquiring CA information for follower catalog manager: success
I20260924 04:38:22.856882 13963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3aa7a7c8ba334fc2ad678854a306ecb8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:38:22.869040 13988 catalog_manager.cc:1259] Loaded cluster ID: d2fa9de2a2dc4043a126a62590b6a43d
I20260924 04:38:22.869107 13988 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1: loading cluster ID for follower catalog manager: success
I20260924 04:38:22.870728 13988 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1: acquiring CA information for follower catalog manager: success
I20260924 04:38:22.871767 13988 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c59fb9b60e244e4f879334cab3670ac1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:38:23.011983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 13716
I20260924 04:38:23.018404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 13791
I20260924 04:38:23.024645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 13867
2026-09-24T04:38:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3003 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:38:23 dist-test-slave-jlnt krb5kdc[14010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:38:23 dist-test-slave-jlnt krb5kdc[14010](info): set up 2 sockets
Sep 24 04:38:23 dist-test-slave-jlnt krb5kdc[14010](info): commencing operation
krb5kdc: starting...
W20260924 04:38:24.073304 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:38:24 dist-test-slave-jlnt krb5kdc[14010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:38:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:38:24.192519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:24.298026 14023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:24.298296 14023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:24.298367 14023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:24.302263 14023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:24.302347 14023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:24.302393 14023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:24.302452 14023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:24.302492 14023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:24.307497 14023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:24.308472 14023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:24.309474 14023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:38:24.317648 14023 server_base.cc:1034] running on GCE node
W20260924 04:38:24.317605 14028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:24.317605 14031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:24.317637 14029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:24.318404 14023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:24.319437 14023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:24.320628 14023 hybrid_clock.cc:648] HybridClock initialized: now 1790224704320615 us; error 26 us; skew 500 ppm
Sep 24 04:38:24 dist-test-slave-jlnt krb5kdc[14010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:24.323349 14023 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:38:24.324112 14023 webserver.cc:460] Webserver started at http://127.0.98.62:40769/ using document root <none> and password file <none>
I20260924 04:38:24.324589 14023 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:24.324645 14023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:24.324793 14023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:24.326756 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "d1ca038b97fd4877baf9ecebd737a445"
format_stamp: "Formatted at 2026-09-24 04:38:24 on dist-test-slave-jlnt"
server_key: "28fbeb08d6db81a638d71bb049485930"
server_key_iv: "f5302997c7f2e2cb6e05daa67f8b031e"
server_key_version: "encryptionkey@0"
I20260924 04:38:24.327256 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1ca038b97fd4877baf9ecebd737a445"
format_stamp: "Formatted at 2026-09-24 04:38:24 on dist-test-slave-jlnt"
server_key: "28fbeb08d6db81a638d71bb049485930"
server_key_iv: "f5302997c7f2e2cb6e05daa67f8b031e"
server_key_version: "encryptionkey@0"
I20260924 04:38:24.330806 14023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 04:38:24.333046 14038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:24.334455 14023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:38:24.334587 14023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "d1ca038b97fd4877baf9ecebd737a445"
format_stamp: "Formatted at 2026-09-24 04:38:24 on dist-test-slave-jlnt"
server_key: "28fbeb08d6db81a638d71bb049485930"
server_key_iv: "f5302997c7f2e2cb6e05daa67f8b031e"
server_key_version: "encryptionkey@0"
I20260924 04:38:24.334690 14023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:24.347275 14023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:24.350468 14023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:24.350663 14023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:24.357053 14023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46061
I20260924 04:38:24.357108 14090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46061 every 8 connection(s)
I20260924 04:38:24.358024 14023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:38:24.360612 14091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:24.365937 14091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445: Bootstrap starting.
I20260924 04:38:24.368279 14091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:24.368290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 14023
I20260924 04:38:24.368403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:38:24.368703 392 external_mini_cluster.cc:1428] Setting key 02d1c122fcf1ab8c12fd319a6362731a
I20260924 04:38:24.369227 14091 log.cc:824] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:24.371152 14091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445: No bootstrap required, opened a new log
I20260924 04:38:24.375308 14091 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ca038b97fd4877baf9ecebd737a445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } }
I20260924 04:38:24.375587 14091 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:24.375644 14091 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1ca038b97fd4877baf9ecebd737a445, State: Initialized, Role: FOLLOWER
I20260924 04:38:24.376096 14091 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ca038b97fd4877baf9ecebd737a445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } }
I20260924 04:38:24.376219 14091 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:38:24.376299 14091 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:38:24.376392 14091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:24.377321 14091 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ca038b97fd4877baf9ecebd737a445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } }
Sep 24 04:38:24 dist-test-slave-jlnt krb5kdc[14010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:24.377645 14091 leader_election.cc:302] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1ca038b97fd4877baf9ecebd737a445; no voters:
I20260924 04:38:24.377986 14091 leader_election.cc:288] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:38:24.378108 14096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:38:24.378371 14096 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [term 1 LEADER]: Becoming Leader. State: Replica: d1ca038b97fd4877baf9ecebd737a445, State: Running, Role: LEADER
I20260924 04:38:24.378978 14096 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ca038b97fd4877baf9ecebd737a445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } }
I20260924 04:38:24.379355 14091 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:24.380914 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1ca038b97fd4877baf9ecebd737a445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ca038b97fd4877baf9ecebd737a445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } } }
I20260924 04:38:24.380988 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1ca038b97fd4877baf9ecebd737a445. Latest consensus state: current_term: 1 leader_uuid: "d1ca038b97fd4877baf9ecebd737a445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ca038b97fd4877baf9ecebd737a445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } } }
I20260924 04:38:24.381050 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:24.381057 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:24.381569 14108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:38:24.383860 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:24.369819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.0.98.62:46061)
0924 04:38:24.370224 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:24.370239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:24.370273 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:38:24.370912 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:24.370928 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:24.371275 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:24.371565 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:24.372423 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:24.373431 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:24.374729 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:24.375186 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:24.375300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:24.378269 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:24.378275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:24.378291 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:24.378322 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:24.380388 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:24.380925 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:24.380931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:24.380937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:24.381088 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:24.381490 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:24.381493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:24.381495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:24.382049 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:24.382274 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:24.382578 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 04:38:24.382911 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":113,"threads_started":1}
I20260924 04:38:24.384720 14108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:38:24.386852 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:38:24.390448 14108 catalog_manager.cc:1347] Generated new cluster ID: 47d084fdd66c4848a98a5a0b81e746f0
I20260924 04:38:24.390519 14108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:38:24.414077 14108 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:38:24.415050 14108 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:38:24.423872 14108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1ca038b97fd4877baf9ecebd737a445: Generated new TSK 0
I20260924 04:38:24.424573 14108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:38:24.442271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 14023
2026-09-24T04:38:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1414 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:38:24 dist-test-slave-jlnt krb5kdc[14118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:38:24 dist-test-slave-jlnt krb5kdc[14118](info): set up 2 sockets
Sep 24 04:38:24 dist-test-slave-jlnt krb5kdc[14118](info): commencing operation
krb5kdc: starting...
W20260924 04:38:25.485527 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:38:25 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:38:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:38:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:38:25.603933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:25.710796 14131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:25.711035 14131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:25.711074 14131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:25.714968 14131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:25.715020 14131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:25.715039 14131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:25.715055 14131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:25.715070 14131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:25.720046 14131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:25.720930 14131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:25.721925 14131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:25.730191 14136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:25.730239 14137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:25.730193 14139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:25.730427 14131 server_base.cc:1034] running on GCE node
I20260924 04:38:25.731081 14131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:25.732100 14131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:25.733266 14131 hybrid_clock.cc:648] HybridClock initialized: now 1790224705733238 us; error 41 us; skew 500 ppm
Sep 24 04:38:25 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:25.735822 14131 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:38:25.736687 14131 webserver.cc:460] Webserver started at http://127.0.98.62:44497/ using document root <none> and password file <none>
I20260924 04:38:25.737267 14131 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:25.737361 14131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:25.737581 14131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:25.740180 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "81e63ac4690f4539b6ee74bb7b215c5d"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "20dc137d61a45603195f58e1dd6ef89e"
server_key_iv: "72aa574f4ffec0da9bc57cdf3389d9e1"
server_key_version: "encryptionkey@0"
I20260924 04:38:25.740939 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "81e63ac4690f4539b6ee74bb7b215c5d"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "20dc137d61a45603195f58e1dd6ef89e"
server_key_iv: "72aa574f4ffec0da9bc57cdf3389d9e1"
server_key_version: "encryptionkey@0"
I20260924 04:38:25.744664 14131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:38:25.747023 14146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:25.748198 14131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:38:25.748335 14131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "81e63ac4690f4539b6ee74bb7b215c5d"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "20dc137d61a45603195f58e1dd6ef89e"
server_key_iv: "72aa574f4ffec0da9bc57cdf3389d9e1"
server_key_version: "encryptionkey@0"
I20260924 04:38:25.748450 14131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:25.766050 14131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:25.768699 14131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:25.768896 14131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:25.775488 14131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44687
I20260924 04:38:25.775768 14198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44687 every 8 connection(s)
I20260924 04:38:25.776443 14131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:38:25.779018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 14131
I20260924 04:38:25.779112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:38:25.779098 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:25.779390 392 external_mini_cluster.cc:1428] Setting key 0af639574b8e7c29337572cbf744d2b4
I20260924 04:38:25.786165 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Bootstrap starting.
I20260924 04:38:25.788504 14199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Neither blocks nor log segments found. Creating new log.
Sep 24 04:38:25 dist-test-slave-jlnt krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:25.789419 14199 log.cc:824] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:25.791432 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: No bootstrap required, opened a new log
I20260924 04:38:25.794049 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:25.780745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.0.98.62:44687)
0924 04:38:25.781205 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:25.781222 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:25.781265 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:38:25.782469 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:25.782494 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:25.782910 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:25.783271 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:25.783956 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:25.785473 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:25.786283 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:25.786924 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:25.787068 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:25.789705 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:25.789710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:25.789721 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:25.789748 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:25.791388 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:25.791855 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:25.791860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:25.791864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:25.791967 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:25.792248 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:25.792250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:25.792251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:25.792607 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:25.792751 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:25.793036 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 04:38:25.793383 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":110,"threads_started":1}
I20260924 04:38:25.795286 14199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:25.795516 14199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:25.795575 14199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e63ac4690f4539b6ee74bb7b215c5d, State: Initialized, Role: FOLLOWER
I20260924 04:38:25.795984 14199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:25.796129 14199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:38:25.796223 14199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:38:25.796343 14199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:25.797223 14199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:25.797545 14199 leader_election.cc:302] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e63ac4690f4539b6ee74bb7b215c5d; no voters:
I20260924 04:38:25.797830 14199 leader_election.cc:288] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:38:25.797974 14204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:38:25.798209 14204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 1 LEADER]: Becoming Leader. State: Replica: 81e63ac4690f4539b6ee74bb7b215c5d, State: Running, Role: LEADER
I20260924 04:38:25.798563 14204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:25.799018 14199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:25.800171 14205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } } }
I20260924 04:38:25.800324 14205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:25.800607 14206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e63ac4690f4539b6ee74bb7b215c5d. Latest consensus state: current_term: 1 leader_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } } }
I20260924 04:38:25.800688 14213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:38:25.800704 14206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:25.803733 14213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:38:25.808933 14213 catalog_manager.cc:1347] Generated new cluster ID: 39079105aa1549f89c91c8a9653aa73f
I20260924 04:38:25.809007 14213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:38:25.829177 14213 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:38:25.830181 14213 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:38:25.835156 14213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Generated new TSK 0
I20260924 04:38:25.835868 14213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:38:25.864035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44687
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:25.972402 14225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:25.972631 14225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:25.972669 14225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:25.976529 14225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:25.976576 14225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:25.976630 14225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:38:25.981623 14225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:25.982487 14225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:25.983371 14225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:25.992537 14231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:25.992600 14225 server_base.cc:1034] running on GCE node
W20260924 04:38:25.992527 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:25.992527 14233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:25.993269 14225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:25.993882 14225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:25.995076 14225 hybrid_clock.cc:648] HybridClock initialized: now 1790224705995036 us; error 54 us; skew 500 ppm
Sep 24 04:38:25 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:25.997798 14225 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:38:25.998592 14225 webserver.cc:460] Webserver started at http://127.0.98.1:41567/ using document root <none> and password file <none>
I20260924 04:38:25.999104 14225 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:25.999159 14225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:25.999423 14225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:26.001590 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "43080799469e4603848160cda44b6c3e"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "f0c904134dbb25b1627c7f1da97d84ca"
server_key_iv: "a0e999f92dc52fb6632667d4ba2db5a7"
server_key_version: "encryptionkey@0"
I20260924 04:38:26.002089 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "43080799469e4603848160cda44b6c3e"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "f0c904134dbb25b1627c7f1da97d84ca"
server_key_iv: "a0e999f92dc52fb6632667d4ba2db5a7"
server_key_version: "encryptionkey@0"
I20260924 04:38:26.005553 14225 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:38:26.007752 14240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:26.008966 14225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:26.009061 14225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "43080799469e4603848160cda44b6c3e"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "f0c904134dbb25b1627c7f1da97d84ca"
server_key_iv: "a0e999f92dc52fb6632667d4ba2db5a7"
server_key_version: "encryptionkey@0"
I20260924 04:38:26.009222 14225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:26.031119 14225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:26.034155 14225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:26.034359 14225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:26.034946 14225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:26.035744 14225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:26.035791 14225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:26.035858 14225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:26.035888 14225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:26.045068 14225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36311
I20260924 04:38:26.045187 14353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36311 every 8 connection(s)
I20260924 04:38:26.046242 14225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:38:26.049830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 14225
I20260924 04:38:26.049957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:38:26.050235 392 external_mini_cluster.cc:1428] Setting key dae32e3967910f9b485655378357aee0
Sep 24 04:38:26 dist-test-slave-jlnt krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:38:26.060496 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.048196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56699 (local address 127.0.98.62:44687)
0924 04:38:26.048344 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.048348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.049347 (+ 999us) server_negotiation.cc:407] Connection header received
0924 04:38:26.050613 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.050617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.050679 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.050773 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:26.052101 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.052751 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.054089 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.054327 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.054441 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.056923 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:26.056926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:26.056928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:26.056951 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:26.058495 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:26.058995 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:26.058999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:26.059001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:26.059083 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:26.059509 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:26.059511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:26.059512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:26.059679 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:26.059755 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.060257 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.060376 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:38:26.061399 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.048507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:56699)
0924 04:38:26.049189 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.049247 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:26.050354 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:26.050940 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:26.050946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:26.051284 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:26.051942 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.051966 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.052879 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.052885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.053961 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.053970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.054474 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.054477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.054656 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.055306 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:26.055331 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:26.056749 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:26.058612 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:26.058618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:26.058632 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:26.058889 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:26.059212 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:26.059215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:26.059217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:26.059408 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:26.059777 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:26.059781 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:26.059999 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 04:38:26.060200 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 04:38:26.060446 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:38:26.062635 14354 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44687
I20260924 04:38:26.062914 14354 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:26.063449 14354 heartbeater.cc:511] Master 127.0.98.62:44687 requested a full tablet report, sending...
I20260924 04:38:26.065047 14163 ts_manager.cc:194] Registered new tserver with Master: 43080799469e4603848160cda44b6c3e (127.0.98.1:36311)
I20260924 04:38:26.066191 14163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:38:26.077804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44687
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:26.185326 14359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:26.185570 14359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:26.185611 14359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:26.189560 14359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:26.189613 14359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:26.189687 14359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:38:26.194753 14359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:26.195674 14359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:26.196597 14359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:38:26.206341 14359 server_base.cc:1034] running on GCE node
W20260924 04:38:26.206266 14364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:26.206266 14367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:26.206272 14365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:26.206930 14359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:26.207551 14359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:26.208743 14359 hybrid_clock.cc:648] HybridClock initialized: now 1790224706208733 us; error 33 us; skew 500 ppm
Sep 24 04:38:26 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:26.211508 14359 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:38:26.212458 14359 webserver.cc:460] Webserver started at http://127.0.98.2:43221/ using document root <none> and password file <none>
I20260924 04:38:26.213177 14359 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:26.213281 14359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:26.213490 14359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:26.215456 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "bbd3cf416da84f9e9907898c53b976c3"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "9ca15f56b042bc8d5bab52f3caac3c88"
server_key_iv: "ae3bc381e23999bf5f567efaa4fcf8d6"
server_key_version: "encryptionkey@0"
I20260924 04:38:26.216012 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bbd3cf416da84f9e9907898c53b976c3"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "9ca15f56b042bc8d5bab52f3caac3c88"
server_key_iv: "ae3bc381e23999bf5f567efaa4fcf8d6"
server_key_version: "encryptionkey@0"
I20260924 04:38:26.219547 14359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:38:26.221876 14374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:26.223119 14359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:38:26.223253 14359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "bbd3cf416da84f9e9907898c53b976c3"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "9ca15f56b042bc8d5bab52f3caac3c88"
server_key_iv: "ae3bc381e23999bf5f567efaa4fcf8d6"
server_key_version: "encryptionkey@0"
I20260924 04:38:26.223357 14359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:26.242956 14359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:26.245756 14359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:26.245956 14359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:26.246513 14359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:26.247283 14359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:26.247350 14359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:26.247421 14359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:26.247462 14359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:26.256102 14359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32945
I20260924 04:38:26.256134 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32945 every 8 connection(s)
I20260924 04:38:26.256922 14359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:38:26.263783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 14359
I20260924 04:38:26.263911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:38:26.264232 392 external_mini_cluster.cc:1428] Setting key b68b757c9a6896a7718178d9e08616a2
Sep 24 04:38:26 dist-test-slave-jlnt krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:26.269964 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.258598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43489 (local address 127.0.98.62:44687)
0924 04:38:26.258737 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.258740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.259683 (+ 943us) server_negotiation.cc:407] Connection header received
0924 04:38:26.260506 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.260509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.260555 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.260626 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:26.261771 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.262292 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.263370 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.263581 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.263659 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.266287 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:26.266291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:26.266294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:26.266319 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:26.267777 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:26.268385 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:26.268388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:26.268389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:26.268464 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:26.268847 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:26.268849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:26.268850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:26.269007 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:26.269084 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.269747 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.269850 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:38:26.271147 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.258943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:43489)
0924 04:38:26.259565 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.259604 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:26.260318 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:26.260769 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:26.260776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:26.261111 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:26.261640 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.261653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.262406 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.262408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.263266 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.263274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.263735 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.263740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.263951 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.264561 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:26.264580 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:26.266126 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:26.267915 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:26.267921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:26.267934 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:26.268283 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:26.268559 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:26.268565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:26.268570 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:26.268753 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:26.269112 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:26.269121 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:26.269462 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 04:38:26.269701 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 04:38:26.270058 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":112,"threads_started":1}
I20260924 04:38:26.272306 14488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44687
I20260924 04:38:26.272564 14488 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:26.273130 14488 heartbeater.cc:511] Master 127.0.98.62:44687 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:38:26.274236 14163 ts_manager.cc:194] Registered new tserver with Master: bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945)
I20260924 04:38:26.274785 14163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:38:26.291600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44687
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:26.398581 14493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:26.398825 14493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:26.398866 14493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:26.402995 14493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:26.403045 14493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:26.403141 14493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:38:26.408171 14493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:26.409112 14493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:26.410132 14493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:26.419633 14499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:26.419747 14493 server_base.cc:1034] running on GCE node
W20260924 04:38:26.419641 14501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:26.419911 14498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:26.420269 14493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:26.420872 14493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:26.422039 14493 hybrid_clock.cc:648] HybridClock initialized: now 1790224706422004 us; error 49 us; skew 500 ppm
Sep 24 04:38:26 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:26.424642 14493 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:38:26.425594 14493 webserver.cc:460] Webserver started at http://127.0.98.3:39063/ using document root <none> and password file <none>
I20260924 04:38:26.426200 14493 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:26.426285 14493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:26.426501 14493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:26.428475 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a9c944515244f95a74840be8b1709e1"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "36e2be089fd855e4a49068c08853fa73"
server_key_iv: "f5ce0c0c85e34f2ef296f723c0856cc8"
server_key_version: "encryptionkey@0"
I20260924 04:38:26.429024 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a9c944515244f95a74840be8b1709e1"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "36e2be089fd855e4a49068c08853fa73"
server_key_iv: "f5ce0c0c85e34f2ef296f723c0856cc8"
server_key_version: "encryptionkey@0"
I20260924 04:38:26.432616 14493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:38:26.434968 14508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:26.436110 14493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:26.436235 14493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "4a9c944515244f95a74840be8b1709e1"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "36e2be089fd855e4a49068c08853fa73"
server_key_iv: "f5ce0c0c85e34f2ef296f723c0856cc8"
server_key_version: "encryptionkey@0"
I20260924 04:38:26.436336 14493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:26.456327 14493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:26.459275 14493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:26.459481 14493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:26.460059 14493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:26.460850 14493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:26.460922 14493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:26.460988 14493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:26.461035 14493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:26.469830 14493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39815
I20260924 04:38:26.469851 14621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39815 every 8 connection(s)
I20260924 04:38:26.470707 14493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:38:26.477432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 14493
I20260924 04:38:26.477550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:38:26.477865 392 external_mini_cluster.cc:1428] Setting key 1cc89422b5f27fce8eba42eaa279d059
Sep 24 04:38:26 dist-test-slave-jlnt krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:26.483987 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.472539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50991 (local address 127.0.98.62:44687)
0924 04:38:26.472686 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.472691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.473929 (+ 1238us) server_negotiation.cc:407] Connection header received
0924 04:38:26.474765 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.474768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.474812 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.474889 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:26.475966 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.476511 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.477746 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.478050 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.478163 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.480709 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:26.480712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:26.480714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:26.480737 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:26.482005 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:26.482483 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:26.482486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:26.482487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:26.482734 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:26.483047 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:26.483049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:26.483051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:26.483233 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:26.483310 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.483749 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.483871 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:38:26.484747 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.473014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:50991)
0924 04:38:26.473778 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.473818 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:26.474573 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:26.475020 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:26.475026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:26.475360 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:26.475829 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.475841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.476615 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.476617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.477612 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.477625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.478209 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.478213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.478427 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.479047 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:26.479062 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:26.480540 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:26.482107 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:26.482112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:26.482122 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:26.482358 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:26.482835 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:26.482837 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:26.482839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:26.482966 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:26.483322 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:26.483327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:26.483554 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 04:38:26.483704 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 04:38:26.483937 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":198,"threads_started":1}
I20260924 04:38:26.486027 14622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44687
I20260924 04:38:26.486258 14622 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:26.486732 14622 heartbeater.cc:511] Master 127.0.98.62:44687 requested a full tablet report, sending...
I20260924 04:38:26.487759 14163 ts_manager.cc:194] Registered new tserver with Master: 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815)
I20260924 04:38:26.488334 14163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50991
I20260924 04:38:26.491446 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:38:26.500738 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.493293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.0.98.62:44687)
0924 04:38:26.493441 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.493445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.493601 (+ 156us) server_negotiation.cc:407] Connection header received
0924 04:38:26.493745 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.493748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.493794 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.493878 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:26.494561 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.495070 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.495903 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.496121 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.496229 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.497554 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:26.497558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:26.497563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:26.497593 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:26.498958 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:26.499405 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:26.499414 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:26.499416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:26.499489 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:26.499827 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:26.499830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:26.499832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:26.500035 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:26.500178 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.500409 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.500517 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:38:26.503443 14163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:38:26.506531 14163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:38:26.520347 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.514532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36311 (local address 127.0.0.1:36458)
0924 04:38:26.514877 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.514921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:26.515037 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:26.516023 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:26.516029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:26.516052 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:26.516189 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.516198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.517421 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.517427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.518792 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.518800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.519834 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.519838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.519944 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.519980 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:38:26.520079 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 04:38:26.520187 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":88,"thread_start_us":88,"threads_started":1}
I20260924 04:38:26.520814 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.514628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.0.98.1:36311)
0924 04:38:26.515546 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.515560 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.515585 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:38:26.515667 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.515674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.515880 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.516032 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:26.516296 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.517284 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.518909 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.519495 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.519608 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.519775 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.520536 (+ 761us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.520646 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":102,"threads_started":1}
I20260924 04:38:26.521466 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.515503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.0.98.3:39815)
0924 04:38:26.515848 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.515860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.516316 (+ 456us) server_negotiation.cc:407] Connection header received
0924 04:38:26.516398 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.516403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.516582 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.516728 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:26.517189 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.517992 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.519632 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.520242 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.520369 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.520524 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.521229 (+ 705us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.521330 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":123,"threads_started":1}
I20260924 04:38:26.521488 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.515311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.98.2:32945)
0924 04:38:26.516340 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.516352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.516374 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:38:26.516433 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.516437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.516581 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.516728 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:26.516983 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.518119 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.519687 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.520679 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.520782 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.520932 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.521032 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.521359 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":141,"threads_started":1}
I20260924 04:38:26.521049 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.515315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39815 (local address 127.0.0.1:60368)
0924 04:38:26.515877 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.515891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:26.515985 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:26.516901 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:26.516904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:26.516923 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:26.517038 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.517045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.518122 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.518128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.519491 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.519502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.520369 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.520372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.520693 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.520712 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:38:26.520810 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:38:26.520882 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":29,"thread_start_us":92,"threads_started":1}
I20260924 04:38:26.521991 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.514623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32945 (local address 127.0.0.1:60376)
0924 04:38:26.515221 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.515234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:26.515324 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:26.516728 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:26.516731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:26.516750 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:26.516855 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.516861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.518242 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.518244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.519559 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.519571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.520809 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.520813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.520904 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.520924 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:38:26.520996 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:38:26.521052 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":143,"thread_start_us":108,"threads_started":1}
I20260924 04:38:26.524122 14288 tablet_service.cc:1467] Processing CreateTablet for tablet ad59ec538d754d848ef3bb7d838f0432 (DEFAULT_TABLE table=test-table [id=6b5fc106ce6f4291b847c8c1abae6185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:26.524792 14422 tablet_service.cc:1467] Processing CreateTablet for tablet ad59ec538d754d848ef3bb7d838f0432 (DEFAULT_TABLE table=test-table [id=6b5fc106ce6f4291b847c8c1abae6185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:26.525300 14288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad59ec538d754d848ef3bb7d838f0432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:26.525820 14422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad59ec538d754d848ef3bb7d838f0432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:26.530923 14556 tablet_service.cc:1467] Processing CreateTablet for tablet ad59ec538d754d848ef3bb7d838f0432 (DEFAULT_TABLE table=test-table [id=6b5fc106ce6f4291b847c8c1abae6185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:26.531705 14556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad59ec538d754d848ef3bb7d838f0432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:26.532650 14639 tablet_bootstrap.cc:492] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Bootstrap starting.
I20260924 04:38:26.533061 14640 tablet_bootstrap.cc:492] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Bootstrap starting.
I20260924 04:38:26.535789 14640 tablet_bootstrap.cc:654] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:26.536650 14640 log.cc:824] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:26.538643 14639 tablet_bootstrap.cc:654] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:26.538684 14640 tablet_bootstrap.cc:492] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: No bootstrap required, opened a new log
I20260924 04:38:26.538877 14640 ts_tablet_manager.cc:1397] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:38:26.538980 14642 tablet_bootstrap.cc:492] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: Bootstrap starting.
I20260924 04:38:26.540691 14639 log.cc:824] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:26.541082 14642 tablet_bootstrap.cc:654] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:26.542116 14642 log.cc:824] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:26.542363 14640 raft_consensus.cc:348] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:26.542639 14640 raft_consensus.cc:374] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:26.542690 14639 tablet_bootstrap.cc:492] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: No bootstrap required, opened a new log
I20260924 04:38:26.542698 14640 raft_consensus.cc:729] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd3cf416da84f9e9907898c53b976c3, State: Initialized, Role: FOLLOWER
I20260924 04:38:26.542856 14639 ts_tablet_manager.cc:1397] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260924 04:38:26.543164 14640 consensus_queue.cc:260] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:26.543776 14488 heartbeater.cc:503] Master 127.0.98.62:44687 was elected leader, sending a full tablet report...
I20260924 04:38:26.544107 14640 ts_tablet_manager.cc:1428] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:38:26.544479 14642 tablet_bootstrap.cc:492] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: No bootstrap required, opened a new log
I20260924 04:38:26.544654 14642 ts_tablet_manager.cc:1397] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:38:26.546687 14639 raft_consensus.cc:348] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:26.546926 14639 raft_consensus.cc:374] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:26.546981 14639 raft_consensus.cc:729] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43080799469e4603848160cda44b6c3e, State: Initialized, Role: FOLLOWER
I20260924 04:38:26.547489 14639 consensus_queue.cc:260] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:26.548146 14354 heartbeater.cc:503] Master 127.0.98.62:44687 was elected leader, sending a full tablet report...
I20260924 04:38:26.548382 14639 ts_tablet_manager.cc:1428] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:38:26.548345 14642 raft_consensus.cc:348] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:26.548601 14642 raft_consensus.cc:374] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:26.548703 14642 raft_consensus.cc:729] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9c944515244f95a74840be8b1709e1, State: Initialized, Role: FOLLOWER
I20260924 04:38:26.549131 14642 consensus_queue.cc:260] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:26.549767 14622 heartbeater.cc:503] Master 127.0.98.62:44687 was elected leader, sending a full tablet report...
I20260924 04:38:26.550010 14642 ts_tablet_manager.cc:1428] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260924 04:38:26.721746 14623 tablet.cc:2369] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:38:26.758010 14489 tablet.cc:2369] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:38:26.792893 14646 raft_consensus.cc:482] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:26.793074 14646 raft_consensus.cc:504] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:26.794269 14646 leader_election.cc:288] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945), 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815)
W20260924 04:38:26.797621 14355 tablet.cc:2369] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:38:26.799628 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.794636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35561 (local address 127.0.98.3:39815)
0924 04:38:26.794891 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.794895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.794913 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:38:26.795056 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.795059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.795113 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.795198 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:26.795570 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.796356 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.798207 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.798813 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.798927 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.798989 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.799424 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.799493 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260924 04:38:26.799626 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.794572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59677 (local address 127.0.98.2:32945)
0924 04:38:26.794796 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.794801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.794895 (+ 94us) server_negotiation.cc:407] Connection header received
0924 04:38:26.795056 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.795059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.795114 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.795199 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:26.795570 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.796346 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.798203 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.798813 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.798927 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.798990 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.799411 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.799489 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260924 04:38:26.799638 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.794534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39815 (local address 127.0.98.1:35561)
0924 04:38:26.794822 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.794837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:26.794941 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:26.795232 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:26.795235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:26.795258 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:26.795462 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.795468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.796475 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.796478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.798072 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.798082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.798983 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.798987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.799213 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.799321 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 04:38:26.799407 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:38:26.799454 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":288,"spinlock_wait_cycles":27136,"thread_start_us":118,"threads_started":1}
I20260924 04:38:26.799641 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.794461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32945 (local address 127.0.98.1:59677)
0924 04:38:26.794791 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.794806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:26.794941 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:26.795232 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:26.795235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:26.795261 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:26.795454 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.795463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.796475 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.796478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.798071 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:26.798082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.798983 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.798987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:26.799208 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.799268 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 04:38:26.799360 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:38:26.799446 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":277,"spinlock_wait_cycles":22656,"thread_start_us":107,"threads_started":1}
I20260924 04:38:26.800348 14442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad59ec538d754d848ef3bb7d838f0432" candidate_uuid: "43080799469e4603848160cda44b6c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd3cf416da84f9e9907898c53b976c3" is_pre_election: true
I20260924 04:38:26.800350 14576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad59ec538d754d848ef3bb7d838f0432" candidate_uuid: "43080799469e4603848160cda44b6c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9c944515244f95a74840be8b1709e1" is_pre_election: true
I20260924 04:38:26.800670 14576 raft_consensus.cc:2454] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43080799469e4603848160cda44b6c3e in term 0.
I20260924 04:38:26.800663 14442 raft_consensus.cc:2454] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43080799469e4603848160cda44b6c3e in term 0.
I20260924 04:38:26.801155 14243 leader_election.cc:302] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43080799469e4603848160cda44b6c3e, 4a9c944515244f95a74840be8b1709e1, bbd3cf416da84f9e9907898c53b976c3; no voters:
I20260924 04:38:26.801405 14646 raft_consensus.cc:2781] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:38:26.801501 14646 raft_consensus.cc:482] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:38:26.801577 14646 raft_consensus.cc:3037] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:26.802562 14646 raft_consensus.cc:504] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:26.802901 14646 leader_election.cc:288] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 1 election: Requested vote from peers bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945), 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815)
I20260924 04:38:26.803208 14576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad59ec538d754d848ef3bb7d838f0432" candidate_uuid: "43080799469e4603848160cda44b6c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9c944515244f95a74840be8b1709e1"
I20260924 04:38:26.803283 14442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad59ec538d754d848ef3bb7d838f0432" candidate_uuid: "43080799469e4603848160cda44b6c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd3cf416da84f9e9907898c53b976c3"
I20260924 04:38:26.803331 14576 raft_consensus.cc:3037] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:26.803423 14442 raft_consensus.cc:3037] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:26.804384 14576 raft_consensus.cc:2454] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43080799469e4603848160cda44b6c3e in term 1.
I20260924 04:38:26.804587 14442 raft_consensus.cc:2454] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43080799469e4603848160cda44b6c3e in term 1.
I20260924 04:38:26.804703 14242 leader_election.cc:302] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43080799469e4603848160cda44b6c3e, 4a9c944515244f95a74840be8b1709e1; no voters:
I20260924 04:38:26.804852 14646 raft_consensus.cc:2781] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:38:26.805063 14646 raft_consensus.cc:686] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 1 LEADER]: Becoming Leader. State: Replica: 43080799469e4603848160cda44b6c3e, State: Running, Role: LEADER
I20260924 04:38:26.805379 14646 consensus_queue.cc:237] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:26.808344 14162 catalog_manager.cc:5495] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 43080799469e4603848160cda44b6c3e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "43080799469e4603848160cda44b6c3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } health_report { overall_health: UNKNOWN } } }
I20260924 04:38:26.883669 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.880172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.0.98.1:36311)
0924 04:38:26.880314 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:26.880319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:26.880334 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:38:26.880463 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:26.880465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:26.880521 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:26.880632 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:38:26.880880 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.881417 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.882262 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:26.882489 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:26.882597 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:26.882903 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:38:26.883294 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:38:26.883436 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:26.883472 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 04:38:26.883524 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:38:26.889413 14442 raft_consensus.cc:1261] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [term 1 FOLLOWER]: Refusing update from remote peer 43080799469e4603848160cda44b6c3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:26.889410 14576 raft_consensus.cc:1261] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [term 1 FOLLOWER]: Refusing update from remote peer 43080799469e4603848160cda44b6c3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:26.890079 14646 consensus_queue.cc:1035] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:26.890280 14650 consensus_queue.cc:1035] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:26.896713 14653 mvcc.cc:205] Tried to move back new op lower bound from 7332760399411617792 to 7332760399077232640. Current Snapshot: MvccSnapshot[applied={T|T < 7332760399411617792}]
I20260924 04:38:26.897203 14655 mvcc.cc:205] Tried to move back new op lower bound from 7332760399411617792 to 7332760399077232640. Current Snapshot: MvccSnapshot[applied={T|T < 7332760399411617792}]
I20260924 04:38:26.904790 14163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:38:26.904951 14163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:38:26.907191 14163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Sending DeleteTablet for 3 replicas of tablet ad59ec538d754d848ef3bb7d838f0432
I20260924 04:38:26.907872 14288 tablet_service.cc:1514] Processing DeleteTablet for tablet ad59ec538d754d848ef3bb7d838f0432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36458
I20260924 04:38:26.907912 14422 tablet_service.cc:1514] Processing DeleteTablet for tablet ad59ec538d754d848ef3bb7d838f0432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60376
I20260924 04:38:26.908324 14662 tablet_replica.cc:330] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: stopping tablet replica
I20260924 04:38:26.908351 14556 tablet_service.cc:1514] Processing DeleteTablet for tablet ad59ec538d754d848ef3bb7d838f0432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60368
I20260924 04:38:26.908586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 14225
I20260924 04:38:26.908627 14662 raft_consensus.cc:2229] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:38:26.908998 14662 raft_consensus.cc:2258] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:26.908833 14664 tablet_replica.cc:330] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: stopping tablet replica
I20260924 04:38:26.909546 14664 raft_consensus.cc:2229] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:38:26.909762 14664 raft_consensus.cc:2258] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:26.910881 14664 ts_tablet_manager.cc:1905] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:38:26.912837 14664 ts_tablet_manager.cc:1918] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:38:26.912912 14664 log.cc:1196] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/ad59ec538d754d848ef3bb7d838f0432
I20260924 04:38:26.913116 14663 tablet_replica.cc:330] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: stopping tablet replica
I20260924 04:38:26.913246 14664 ts_tablet_manager.cc:1939] T ad59ec538d754d848ef3bb7d838f0432 P 4a9c944515244f95a74840be8b1709e1: Deleting consensus metadata
I20260924 04:38:26.913468 14663 raft_consensus.cc:2229] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:38:26.913760 14663 raft_consensus.cc:2258] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:26.914408 14148 catalog_manager.cc:4841] TS 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815): tablet ad59ec538d754d848ef3bb7d838f0432 (table test-table [id=6b5fc106ce6f4291b847c8c1abae6185]) successfully deleted
I20260924 04:38:26.915191 14663 ts_tablet_manager.cc:1905] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:38:26.916500 14150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36311 (error 108)
I20260924 04:38:26.916630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 14359
I20260924 04:38:26.917936 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.917689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36311 (local address 127.0.0.1:36468)
0924 04:38:26.917807 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.917870 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 04:38:26.918080 14150 catalog_manager.cc:4584] TS 43080799469e4603848160cda44b6c3e (127.0.98.1:36311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad59ec538d754d848ef3bb7d838f0432: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36311: connect: Connection refused (error 111)
I20260924 04:38:26.923111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 14493
I20260924 04:38:26.924727 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:26.923270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32945 (local address 127.0.0.1:60382)
0924 04:38:26.924615 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:26.924663 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1275}
I20260924 04:38:26.929332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 14131
I20260924 04:38:26.936017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44687
--webserver_interface=127.0.98.62
--webserver_port=44497
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:27.044327 14666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:27.044586 14666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:27.044625 14666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:27.049393 14666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:27.049469 14666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:27.049503 14666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:27.049532 14666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:27.049566 14666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:27.054996 14666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:27.055986 14666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:27.056978 14666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:27.069772 14672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:27.069772 14673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:27.069772 14675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:27.069820 14666 server_base.cc:1034] running on GCE node
I20260924 04:38:27.070652 14666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:27.071812 14666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:27.073011 14666 hybrid_clock.cc:648] HybridClock initialized: now 1790224707072993 us; error 30 us; skew 500 ppm
Sep 24 04:38:27 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:27.075817 14666 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:38:27.076742 14666 webserver.cc:460] Webserver started at http://127.0.98.62:44497/ using document root <none> and password file <none>
I20260924 04:38:27.077382 14666 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:27.077493 14666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:27.081348 14666 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 04:38:27.083521 14682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:27.084838 14666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:27.084998 14666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "81e63ac4690f4539b6ee74bb7b215c5d"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "20dc137d61a45603195f58e1dd6ef89e"
server_key_iv: "72aa574f4ffec0da9bc57cdf3389d9e1"
server_key_version: "encryptionkey@0"
I20260924 04:38:27.085670 14666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:27.100852 14666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:27.103638 14666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:27.103845 14666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:27.110734 14666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44687
I20260924 04:38:27.111027 14734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44687 every 8 connection(s)
I20260924 04:38:27.111855 14666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:38:27.115947 14735 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:38:27.117581 14735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Bootstrap starting.
I20260924 04:38:27.122017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 14666
I20260924 04:38:27.122426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36311
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44687
--webserver_port=41567
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:38:27.134040 14735 log.cc:824] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:27.142197 14735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:38:27.142557 14735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Bootstrap complete.
I20260924 04:38:27.146517 14735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:27.146911 14735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e63ac4690f4539b6ee74bb7b215c5d, State: Initialized, Role: FOLLOWER
I20260924 04:38:27.147365 14735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:27.147508 14735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:38:27.147594 14735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:38:27.147713 14735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:38:27.149546 14735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:27.149879 14735 leader_election.cc:302] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e63ac4690f4539b6ee74bb7b215c5d; no voters:
I20260924 04:38:27.150157 14735 leader_election.cc:288] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:38:27.150303 14739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:38:27.150555 14739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 2 LEADER]: Becoming Leader. State: Replica: 81e63ac4690f4539b6ee74bb7b215c5d, State: Running, Role: LEADER
I20260924 04:38:27.150849 14739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:27.151289 14735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:27.152236 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } } }
I20260924 04:38:27.152356 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:27.153070 14741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e63ac4690f4539b6ee74bb7b215c5d. Latest consensus state: current_term: 2 leader_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } } }
I20260924 04:38:27.153190 14741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:27.156705 14754 catalog_manager.cc:1259] Loaded cluster ID: 39079105aa1549f89c91c8a9653aa73f
I20260924 04:38:27.156764 14754 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: loading cluster ID for follower catalog manager: success
W20260924 04:38:27.157760 14754 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260924 04:38:27.157835 14754 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260924 04:38:27.157907 14748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:38:27.160269 14748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:38:27.160722 14748 catalog_manager.cc:1259] Loaded cluster ID: 39079105aa1549f89c91c8a9653aa73f
I20260924 04:38:27.160777 14748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:38:27.162202 14748 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 04:38:27.166648 14748 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 04:38:27.172709 14748 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:38:27.173826 14748 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Loaded TSK: 0
I20260924 04:38:27.175649 14748 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:38:27.240442 14737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:27.240679 14737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:27.240720 14737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:27.244792 14737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:27.244845 14737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:27.244900 14737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:38:27.249971 14737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41567
--tserver_master_addrs=127.0.98.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:27.250887 14737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:27.251869 14737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:27.261026 14763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:27.261030 14762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:27.261026 14765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:27.261518 14737 server_base.cc:1034] running on GCE node
I20260924 04:38:27.261885 14737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:27.262487 14737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:27.263669 14737 hybrid_clock.cc:648] HybridClock initialized: now 1790224707263647 us; error 41 us; skew 500 ppm
Sep 24 04:38:27 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:27.266366 14737 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:38:27.267271 14737 webserver.cc:460] Webserver started at http://127.0.98.1:41567/ using document root <none> and password file <none>
I20260924 04:38:27.267793 14737 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:27.267866 14737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:27.271646 14737 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:38:27.273705 14772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:27.274942 14737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:27.275060 14737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "43080799469e4603848160cda44b6c3e"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "f0c904134dbb25b1627c7f1da97d84ca"
server_key_iv: "a0e999f92dc52fb6632667d4ba2db5a7"
server_key_version: "encryptionkey@0"
I20260924 04:38:27.275570 14737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:27.309599 14737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:27.313388 14737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:27.313597 14737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:27.314420 14737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:27.316013 14779 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:38:27.321440 14737 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:38:27.321497 14737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 04:38:27.321579 14737 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:38:27.323269 14737 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:38:27.323339 14737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:38:27.323633 14779 tablet_bootstrap.cc:492] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Bootstrap starting.
I20260924 04:38:27.331769 14737 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36311
I20260924 04:38:27.331849 14886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36311 every 8 connection(s)
I20260924 04:38:27.332672 14737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:38:27.338811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 14737
I20260924 04:38:27.339219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32945
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44687
--webserver_port=43221
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:38:27.343896 14779 log.cc:824] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:38:27 dist-test-slave-jlnt krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:27.349932 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.334784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35003 (local address 127.0.98.62:44687)
0924 04:38:27.335276 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.335294 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.335342 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:38:27.336556 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.336571 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.336959 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.337409 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:27.338371 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.340111 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.341487 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.341941 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.342112 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.345038 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:27.345042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:27.345054 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:27.345079 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:27.346855 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:27.347373 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:27.347386 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:27.347390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:27.347498 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:27.347851 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:27.347854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:27.347856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:27.348175 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:27.348313 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.348826 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.349070 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":128,"threads_started":1}
I20260924 04:38:27.349946 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.334732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:35003)
0924 04:38:27.335177 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:27.335227 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:27.336107 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:27.337407 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:27.337413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:27.337767 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:27.338205 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.338217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.340301 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.340308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.341357 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.341367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.342099 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.342103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.342301 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.342906 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:27.342930 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:27.344830 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:27.346982 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:27.346987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:27.347000 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:27.347258 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:27.347596 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:27.347600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:27.347602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:27.347750 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:27.348300 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:27.348305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:27.348534 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 04:38:27.348711 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 04:38:27.349023 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":131,"threads_started":1}
I20260924 04:38:27.352519 14887 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44687
I20260924 04:38:27.352741 14887 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:27.353261 14887 heartbeater.cc:511] Master 127.0.98.62:44687 requested a full tablet report, sending...
I20260924 04:38:27.353670 14779 tablet_bootstrap.cc:492] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:38:27.354043 14779 tablet_bootstrap.cc:492] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Bootstrap complete.
I20260924 04:38:27.355674 14779 ts_tablet_manager.cc:1397] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260924 04:38:27.355717 14699 ts_manager.cc:194] Registered new tserver with Master: 43080799469e4603848160cda44b6c3e (127.0.98.1:36311)
I20260924 04:38:27.358494 14779 raft_consensus.cc:348] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:27.358942 14699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35003
I20260924 04:38:27.358990 14779 raft_consensus.cc:729] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43080799469e4603848160cda44b6c3e, State: Initialized, Role: FOLLOWER
I20260924 04:38:27.359443 14779 consensus_queue.cc:260] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } }
I20260924 04:38:27.360437 14779 ts_tablet_manager.cc:1428] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:38:27.361641 14887 heartbeater.cc:503] Master 127.0.98.62:44687 was elected leader, sending a full tablet report...
I20260924 04:38:27.364764 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.358923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.0.98.1:36311)
0924 04:38:27.359413 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.359424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.360265 (+ 841us) server_negotiation.cc:407] Connection header received
0924 04:38:27.360327 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.360332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.360791 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.360933 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:27.361361 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.362240 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.363624 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.364128 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.364259 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.364380 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.364558 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.364641 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":104,"threads_started":1}
I20260924 04:38:27.364771 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.358277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36311 (local address 127.0.0.1:36476)
0924 04:38:27.359977 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:27.360020 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:27.360139 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:27.361067 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:27.361072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:27.361093 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:27.361234 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.361241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.362368 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.362372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.363504 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.363513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.364260 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.364264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.364368 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.364407 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:38:27.364531 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 04:38:27.364645 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1540,"thread_start_us":95,"threads_started":1}
I20260924 04:38:27.365729 14809 tablet_service.cc:1514] Processing DeleteTablet for tablet ad59ec538d754d848ef3bb7d838f0432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36476
I20260924 04:38:27.366576 14896 tablet_replica.cc:330] stopping tablet replica
I20260924 04:38:27.366865 14896 raft_consensus.cc:2229] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:38:27.367084 14896 raft_consensus.cc:2258] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:27.367910 14896 ts_tablet_manager.cc:1905] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:38:27.371246 14896 ts_tablet_manager.cc:1918] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:38:27.371357 14896 log.cc:1196] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/ad59ec538d754d848ef3bb7d838f0432
I20260924 04:38:27.371618 14896 ts_tablet_manager.cc:1939] T ad59ec538d754d848ef3bb7d838f0432 P 43080799469e4603848160cda44b6c3e: Deleting consensus metadata
I20260924 04:38:27.373118 14686 catalog_manager.cc:4841] TS 43080799469e4603848160cda44b6c3e (127.0.98.1:36311): tablet ad59ec538d754d848ef3bb7d838f0432 (table test-table [id=6b5fc106ce6f4291b847c8c1abae6185]) successfully deleted
W20260924 04:38:27.456470 14891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:27.456707 14891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:27.456746 14891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:27.460745 14891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:27.460799 14891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:27.460856 14891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:38:27.465983 14891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43221
--tserver_master_addrs=127.0.98.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:27.467002 14891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:27.468003 14891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:27.477253 14904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:27.477231 14901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:27.477233 14902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:27.477460 14891 server_base.cc:1034] running on GCE node
I20260924 04:38:27.477948 14891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:27.478628 14891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:27.479808 14891 hybrid_clock.cc:648] HybridClock initialized: now 1790224707479789 us; error 29 us; skew 500 ppm
Sep 24 04:38:27 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:27.482470 14891 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:38:27.483330 14891 webserver.cc:460] Webserver started at http://127.0.98.2:43221/ using document root <none> and password file <none>
I20260924 04:38:27.483879 14891 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:27.483981 14891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:27.487756 14891 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:38:27.490097 14911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:27.491261 14891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:38:27.491402 14891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "bbd3cf416da84f9e9907898c53b976c3"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "9ca15f56b042bc8d5bab52f3caac3c88"
server_key_iv: "ae3bc381e23999bf5f567efaa4fcf8d6"
server_key_version: "encryptionkey@0"
I20260924 04:38:27.491957 14891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:27.507900 14891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:27.510473 14891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:27.510663 14891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:27.511198 14891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:27.512185 14918 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 04:38:27.515693 14918 ts_tablet_manager.cc:1852] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 04:38:27.516064 14918 ts_tablet_manager.cc:1905] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:38:27.519693 14918 ts_tablet_manager.cc:1918] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:38:27.519785 14918 log.cc:1196] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/ad59ec538d754d848ef3bb7d838f0432
I20260924 04:38:27.520044 14918 ts_tablet_manager.cc:1939] T ad59ec538d754d848ef3bb7d838f0432 P bbd3cf416da84f9e9907898c53b976c3: Deleting consensus metadata
I20260924 04:38:27.520287 14891 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 04:38:27.520364 14891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260924 04:38:27.520458 14891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:27.520505 14891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:27.529666 14891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32945
I20260924 04:38:27.529701 15025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32945 every 8 connection(s)
I20260924 04:38:27.530505 14891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:38:27.537292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 14891
I20260924 04:38:27.537671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39815
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44687
--webserver_port=39063
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:38:27 dist-test-slave-jlnt krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:27.544195 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.532330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55649 (local address 127.0.98.62:44687)
0924 04:38:27.532476 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.532480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.533287 (+ 807us) server_negotiation.cc:407] Connection header received
0924 04:38:27.534202 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.534205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.534267 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.534375 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:27.535504 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.536054 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.537173 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.537467 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.537580 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.540091 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:27.540094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:27.540096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:27.540122 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:27.541787 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:27.542342 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:27.542348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:27.542351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:27.542448 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:27.542843 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:27.542849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:27.542852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:27.543111 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:27.543226 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.543770 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.543921 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:38:27.544786 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.532670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:55649)
0924 04:38:27.533090 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:27.533171 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:27.533992 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:27.534540 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:27.534547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:27.534902 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:27.535350 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.535362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.536183 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.536186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.537030 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.537038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.537615 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.537620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.537833 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.538510 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:27.538524 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:27.539919 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:27.541934 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:27.541943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:27.541959 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:27.542224 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:27.542554 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:27.542556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:27.542557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:27.542743 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:27.543253 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:27.543258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:27.543501 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 04:38:27.543695 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 04:38:27.543942 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260924 04:38:27.546061 15026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44687
I20260924 04:38:27.546331 15026 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:27.546944 15026 heartbeater.cc:511] Master 127.0.98.62:44687 requested a full tablet report, sending...
I20260924 04:38:27.548151 14699 ts_manager.cc:194] Registered new tserver with Master: bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945)
I20260924 04:38:27.548763 14699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55649
W20260924 04:38:27.647714 15029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:27.647959 15029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:27.648000 15029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:27.652303 15029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:27.652361 15029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:27.652447 15029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:38:27.657756 15029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39063
--tserver_master_addrs=127.0.98.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:27.658757 15029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:27.659725 15029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:27.669394 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:27.669394 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:27.669559 15029 server_base.cc:1034] running on GCE node
W20260924 04:38:27.669541 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:27.670166 15029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:27.670826 15029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:27.671985 15029 hybrid_clock.cc:648] HybridClock initialized: now 1790224707671957 us; error 42 us; skew 500 ppm
Sep 24 04:38:27 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:27.675138 15029 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:38:27.675941 15029 webserver.cc:460] Webserver started at http://127.0.98.3:39063/ using document root <none> and password file <none>
I20260924 04:38:27.676461 15029 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:27.676532 15029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:27.680383 15029 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:27.682426 15044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:27.683736 15029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:38:27.683849 15029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "4a9c944515244f95a74840be8b1709e1"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "36e2be089fd855e4a49068c08853fa73"
server_key_iv: "f5ce0c0c85e34f2ef296f723c0856cc8"
server_key_version: "encryptionkey@0"
I20260924 04:38:27.684455 15029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:27.705062 15029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:27.708263 15029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:27.708478 15029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:27.709066 15029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:27.709906 15029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:27.709952 15029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:27.710016 15029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:27.710049 15029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:27.718950 15029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39815
I20260924 04:38:27.718973 15157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39815 every 8 connection(s)
I20260924 04:38:27.719923 15029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:38:27.724860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 15029
Sep 24 04:38:27 dist-test-slave-jlnt krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:27.733687 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.725593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.0.98.62:44687)
0924 04:38:27.725868 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.725872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.725964 (+ 92us) server_negotiation.cc:407] Connection header received
0924 04:38:27.726051 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.726054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.726099 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.726193 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:27.726854 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.727512 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.728348 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.728688 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.728793 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.730238 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:27.730243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:27.730246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:27.730272 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:27.732043 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:27.732460 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:27.732465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:27.732467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:27.732552 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:27.732821 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:27.732825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:27.732827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:27.733004 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:27.733251 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.733378 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.733508 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":65,"threads_started":1}
I20260924 04:38:27.734418 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.721753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57593 (local address 127.0.98.62:44687)
0924 04:38:27.721915 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.721919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.722919 (+ 1000us) server_negotiation.cc:407] Connection header received
0924 04:38:27.723724 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.723727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.723773 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.723846 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:27.724983 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.725723 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.727122 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.727332 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.727419 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.730359 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:27.730363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:27.730364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:27.730397 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:27.732117 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:27.732614 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:27.732617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:27.732619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:27.732697 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:27.733028 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:27.733033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:27.733034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:27.733264 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:27.733392 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.734167 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.734270 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:38:27.735422 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.722125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:57593)
0924 04:38:27.722743 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:27.722781 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:27.723552 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:27.724024 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:27.724030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:27.724365 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:27.724825 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.724844 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.725847 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.725850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.727000 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.727010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.727471 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.727475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.727681 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.728269 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:27.728285 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:27.730160 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:27.732232 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:27.732238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:27.732252 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:27.732504 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:27.732786 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:27.732790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:27.732792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:27.732935 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:27.733395 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:27.733403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:27.733688 (+ 285us) client_negotiation.cc:769] Sending connection context
0924 04:38:27.734129 (+ 441us) client_negotiation.cc:240] Negotiation successful
0924 04:38:27.734438 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":111,"threads_started":1}
I20260924 04:38:27.736405 15158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44687
I20260924 04:38:27.736616 15158 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:27.737037 15158 heartbeater.cc:511] Master 127.0.98.62:44687 requested a full tablet report, sending...
I20260924 04:38:27.738013 14699 ts_manager.cc:194] Registered new tserver with Master: 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815)
I20260924 04:38:27.738526 14699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57593
I20260924 04:38:27.747074 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:38:27.751628 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.747693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.0.98.62:44687)
0924 04:38:27.747847 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.747854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.748043 (+ 189us) server_negotiation.cc:407] Connection header received
0924 04:38:27.748242 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.748244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.748348 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.748467 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:38:27.748701 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.749282 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.750192 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.750420 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.750533 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.750849 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:38:27.751247 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:38:27.751386 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.751431 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.751484 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:38:27.752267 14699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:38:27.755813 14699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:38:27.763880 14809 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5a2b77dccf47ab8726f08450ff471d (DEFAULT_TABLE table=test-table [id=532fb79002cb47328ad22622515d5cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:27.764583 14809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5a2b77dccf47ab8726f08450ff471d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:27.767501 14779 tablet_bootstrap.cc:492] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Bootstrap starting.
I20260924 04:38:27.768416 14779 tablet_bootstrap.cc:654] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:27.768505 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.762493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39815 (local address 127.0.0.1:60384)
0924 04:38:27.762600 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:27.762615 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:27.762693 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:27.763774 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:27.763777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:27.763795 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:27.763910 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.763920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.765422 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.765425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.766769 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.766780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.768021 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.768024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.768135 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.768186 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 04:38:27.768262 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:38:27.768371 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":167}
I20260924 04:38:27.768675 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.763084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.0.98.2:32945)
0924 04:38:27.763482 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.763492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.763512 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:38:27.763579 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.763584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.763748 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.763875 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:27.764279 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.765403 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.766863 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.767613 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.767717 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.767885 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.768414 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.768503 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":94,"threads_started":1}
I20260924 04:38:27.768953 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.762886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.0.98.3:39815)
0924 04:38:27.763244 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.763259 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.763283 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:38:27.763366 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.763372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.763617 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.763787 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:27.764017 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.765285 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.766962 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.767910 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.768022 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.768153 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.768294 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.768413 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":149,"threads_started":1}
I20260924 04:38:27.768970 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.762901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32945 (local address 127.0.0.1:60390)
0924 04:38:27.763344 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:27.763358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:27.763450 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:27.764045 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:27.764049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:27.764070 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:27.764151 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.764156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.765519 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.765523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.766701 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.766717 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.767949 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.767952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.768075 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.768189 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 04:38:27.768261 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:38:27.768818 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":477,"thread_start_us":103,"threads_started":1}
I20260924 04:38:27.769668 14779 tablet_bootstrap.cc:492] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: No bootstrap required, opened a new log
I20260924 04:38:27.769780 14779 ts_tablet_manager.cc:1397] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:27.770511 14779 raft_consensus.cc:348] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:27.770640 14779 raft_consensus.cc:374] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:27.770679 14779 raft_consensus.cc:729] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43080799469e4603848160cda44b6c3e, State: Initialized, Role: FOLLOWER
I20260924 04:38:27.770748 14960 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5a2b77dccf47ab8726f08450ff471d (DEFAULT_TABLE table=test-table [id=532fb79002cb47328ad22622515d5cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:27.771530 14779 consensus_queue.cc:260] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:27.771899 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5a2b77dccf47ab8726f08450ff471d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:27.772269 14779 ts_tablet_manager.cc:1428] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:38:27.773741 15092 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5a2b77dccf47ab8726f08450ff471d (DEFAULT_TABLE table=test-table [id=532fb79002cb47328ad22622515d5cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:27.774842 15092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5a2b77dccf47ab8726f08450ff471d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:27.776639 14918 tablet_bootstrap.cc:492] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: Bootstrap starting.
I20260924 04:38:27.778928 14918 tablet_bootstrap.cc:654] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:27.779803 14918 log.cc:824] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:27.781870 14918 tablet_bootstrap.cc:492] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: No bootstrap required, opened a new log
I20260924 04:38:27.782066 14918 ts_tablet_manager.cc:1397] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:38:27.782487 15172 tablet_bootstrap.cc:492] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: Bootstrap starting.
I20260924 04:38:27.784720 15172 tablet_bootstrap.cc:654] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:27.785173 14918 raft_consensus.cc:348] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:27.785442 14918 raft_consensus.cc:374] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:27.785504 14918 raft_consensus.cc:729] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd3cf416da84f9e9907898c53b976c3, State: Initialized, Role: FOLLOWER
I20260924 04:38:27.785612 15172 log.cc:824] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:27.785953 14918 consensus_queue.cc:260] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:27.786616 15026 heartbeater.cc:503] Master 127.0.98.62:44687 was elected leader, sending a full tablet report...
I20260924 04:38:27.786914 14918 ts_tablet_manager.cc:1428] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:38:27.787621 15172 tablet_bootstrap.cc:492] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: No bootstrap required, opened a new log
I20260924 04:38:27.787842 15172 ts_tablet_manager.cc:1397] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:38:27.791208 15172 raft_consensus.cc:348] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:27.791473 15172 raft_consensus.cc:374] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:27.791589 15172 raft_consensus.cc:729] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9c944515244f95a74840be8b1709e1, State: Initialized, Role: FOLLOWER
I20260924 04:38:27.792053 15172 consensus_queue.cc:260] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:27.792769 15158 heartbeater.cc:503] Master 127.0.98.62:44687 was elected leader, sending a full tablet report...
I20260924 04:38:27.793059 15172 ts_tablet_manager.cc:1428] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 04:38:27.833873 14888 tablet.cc:2369] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:38:27.889014 15174 raft_consensus.cc:482] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:27.889317 15174 raft_consensus.cc:504] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:27.890641 15174 leader_election.cc:288] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815), 43080799469e4603848160cda44b6c3e (127.0.98.1:36311)
I20260924 04:38:27.895246 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.890924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36311 (local address 127.0.98.2:58285)
0924 04:38:27.891130 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:27.891146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:27.891245 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:27.891667 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:27.891673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:27.891693 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:27.891844 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.891849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.892602 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.892605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.894018 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.894029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.894733 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.894736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.894864 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.894911 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 04:38:27.895002 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:38:27.895079 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":73}
I20260924 04:38:27.895246 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.890968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50655 (local address 127.0.98.3:39815)
0924 04:38:27.891112 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.891118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.891363 (+ 245us) server_negotiation.cc:407] Connection header received
0924 04:38:27.891553 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.891556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.891624 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.891700 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:27.892035 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.892592 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.894191 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.894583 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.894700 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.894763 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.895061 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.895137 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:38:27.895447 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.890926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39815 (local address 127.0.98.2:50655)
0924 04:38:27.891273 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:27.891288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:27.891440 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:27.891751 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:27.891754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:27.891771 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:27.891920 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.891929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.892700 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.892702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.894029 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:27.894040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.894708 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.894711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:27.894844 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.894896 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 04:38:27.895039 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 04:38:27.895102 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":23,"thread_start_us":135,"threads_started":1}
I20260924 04:38:27.895627 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.890992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58285 (local address 127.0.98.1:36311)
0924 04:38:27.891394 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.891398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.891412 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:38:27.891469 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.891475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.891531 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.891614 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:27.891970 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.892504 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.894160 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.894591 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.894715 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.894790 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.895422 (+ 632us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.895503 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":94,"threads_started":1}
I20260924 04:38:27.896157 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5a2b77dccf47ab8726f08450ff471d" candidate_uuid: "bbd3cf416da84f9e9907898c53b976c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43080799469e4603848160cda44b6c3e" is_pre_election: true
I20260924 04:38:27.896212 15112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5a2b77dccf47ab8726f08450ff471d" candidate_uuid: "bbd3cf416da84f9e9907898c53b976c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9c944515244f95a74840be8b1709e1" is_pre_election: true
I20260924 04:38:27.896492 14841 raft_consensus.cc:2454] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbd3cf416da84f9e9907898c53b976c3 in term 0.
I20260924 04:38:27.896492 15112 raft_consensus.cc:2454] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbd3cf416da84f9e9907898c53b976c3 in term 0.
I20260924 04:38:27.897004 14913 leader_election.cc:302] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9c944515244f95a74840be8b1709e1, bbd3cf416da84f9e9907898c53b976c3; no voters:
I20260924 04:38:27.897326 15174 raft_consensus.cc:2781] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:38:27.897421 15174 raft_consensus.cc:482] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:38:27.897468 15174 raft_consensus.cc:3037] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:27.898538 15174 raft_consensus.cc:504] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:27.898906 15174 leader_election.cc:288] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815), 43080799469e4603848160cda44b6c3e (127.0.98.1:36311)
I20260924 04:38:27.899272 15112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5a2b77dccf47ab8726f08450ff471d" candidate_uuid: "bbd3cf416da84f9e9907898c53b976c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9c944515244f95a74840be8b1709e1"
I20260924 04:38:27.899272 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5a2b77dccf47ab8726f08450ff471d" candidate_uuid: "bbd3cf416da84f9e9907898c53b976c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43080799469e4603848160cda44b6c3e"
I20260924 04:38:27.899405 14841 raft_consensus.cc:3037] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:27.899411 15112 raft_consensus.cc:3037] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:27.900687 14841 raft_consensus.cc:2454] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbd3cf416da84f9e9907898c53b976c3 in term 1.
I20260924 04:38:27.900687 15112 raft_consensus.cc:2454] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbd3cf416da84f9e9907898c53b976c3 in term 1.
I20260924 04:38:27.901001 14915 leader_election.cc:302] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43080799469e4603848160cda44b6c3e, bbd3cf416da84f9e9907898c53b976c3; no voters:
I20260924 04:38:27.901253 15174 raft_consensus.cc:2781] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:38:27.901473 15174 raft_consensus.cc:686] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 1 LEADER]: Becoming Leader. State: Replica: bbd3cf416da84f9e9907898c53b976c3, State: Running, Role: LEADER
I20260924 04:38:27.901870 15174 consensus_queue.cc:237] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:27.905002 14698 catalog_manager.cc:5495] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bbd3cf416da84f9e9907898c53b976c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } health_report { overall_health: UNKNOWN } } }
I20260924 04:38:27.920940 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:27.917695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.98.2:32945)
0924 04:38:27.917836 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:27.917840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:27.917854 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:38:27.917947 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:27.917949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:27.918001 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:27.918096 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:38:27.918329 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.918811 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.919598 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:27.919802 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:27.919877 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:27.920173 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:38:27.920576 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:38:27.920664 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:27.920767 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 04:38:27.920815 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:38:27.927327 14841 raft_consensus.cc:1261] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Refusing update from remote peer bbd3cf416da84f9e9907898c53b976c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:27.927330 15112 raft_consensus.cc:1261] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [term 1 FOLLOWER]: Refusing update from remote peer bbd3cf416da84f9e9907898c53b976c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:27.928081 15174 consensus_queue.cc:1035] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:27.928416 15178 consensus_queue.cc:1035] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:27.936328 15182 mvcc.cc:205] Tried to move back new op lower bound from 7332760403662274560 to 7332760403568726016. Current Snapshot: MvccSnapshot[applied={T|T < 7332760403662274560 or (T in {7332760403662274560})}]
I20260924 04:38:27.943575 14698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:38:27.943758 14698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:38:27.946085 14698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Sending DeleteTablet for 3 replicas of tablet 9c5a2b77dccf47ab8726f08450ff471d
I20260924 04:38:27.946831 14809 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5a2b77dccf47ab8726f08450ff471d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36476
I20260924 04:38:27.946815 15092 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5a2b77dccf47ab8726f08450ff471d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60384
I20260924 04:38:27.946851 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5a2b77dccf47ab8726f08450ff471d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260924 04:38:27.947283 15192 tablet_replica.cc:330] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: stopping tablet replica
I20260924 04:38:27.947347 15193 tablet_replica.cc:330] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: stopping tablet replica
I20260924 04:38:27.947378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 14737
I20260924 04:38:27.947633 15193 raft_consensus.cc:2229] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:38:27.948009 15193 raft_consensus.cc:2258] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:27.949466 15193 ts_tablet_manager.cc:1905] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:38:27.952337 15193 ts_tablet_manager.cc:1918] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:38:27.952409 15193 log.cc:1196] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/9c5a2b77dccf47ab8726f08450ff471d
I20260924 04:38:27.952682 15193 ts_tablet_manager.cc:1939] T 9c5a2b77dccf47ab8726f08450ff471d P bbd3cf416da84f9e9907898c53b976c3: Deleting consensus metadata
W20260924 04:38:27.953804 14686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36311 (error 108)
I20260924 04:38:27.954437 14685 catalog_manager.cc:4841] TS bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945): tablet 9c5a2b77dccf47ab8726f08450ff471d (table test-table [id=532fb79002cb47328ad22622515d5cc6]) successfully deleted
W20260924 04:38:27.955243 15167 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:38:27.954607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36311 (local address 127.0.0.1:36484)
0924 04:38:27.954744 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:27.954761 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:27.954853 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:27.955138 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 04:38:27.955365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 14891
W20260924 04:38:27.955416 14686 catalog_manager.cc:4584] TS 43080799469e4603848160cda44b6c3e (127.0.98.1:36311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c5a2b77dccf47ab8726f08450ff471d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:38:27.957566 15192 raft_consensus.cc:2229] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:38:27.957924 15192 raft_consensus.cc:2258] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:27.959311 15192 ts_tablet_manager.cc:1905] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:38:27.961522 15192 ts_tablet_manager.cc:1918] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:38:27.961632 15192 log.cc:1196] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/9c5a2b77dccf47ab8726f08450ff471d
I20260924 04:38:27.961975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 15029
I20260924 04:38:27.962006 15192 ts_tablet_manager.cc:1939] T 9c5a2b77dccf47ab8726f08450ff471d P 4a9c944515244f95a74840be8b1709e1: Deleting consensus metadata
I20260924 04:38:27.968237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 14666
I20260924 04:38:27.974920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44687
--webserver_interface=127.0.98.62
--webserver_port=44497
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:28.084456 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:28.084718 15195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:28.084759 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:28.089044 15195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:28.089102 15195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:28.089121 15195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:28.089172 15195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:28.089202 15195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:28.094676 15195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:28.095691 15195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:28.096726 15195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:38:28.105376 15195 server_base.cc:1034] running on GCE node
W20260924 04:38:28.105335 15201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:28.105360 15200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:28.105597 15203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:28.106158 15195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:28.107331 15195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:28.108534 15195 hybrid_clock.cc:648] HybridClock initialized: now 1790224708108518 us; error 29 us; skew 500 ppm
Sep 24 04:38:28 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:28.111428 15195 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:38:28.112327 15195 webserver.cc:460] Webserver started at http://127.0.98.62:44497/ using document root <none> and password file <none>
I20260924 04:38:28.112885 15195 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:28.112979 15195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:28.116693 15195 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:38:28.118764 15210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:28.120000 15195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:38:28.120142 15195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "81e63ac4690f4539b6ee74bb7b215c5d"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "20dc137d61a45603195f58e1dd6ef89e"
server_key_iv: "72aa574f4ffec0da9bc57cdf3389d9e1"
server_key_version: "encryptionkey@0"
I20260924 04:38:28.120682 15195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:28.143414 15195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:28.146627 15195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:28.146831 15195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:28.153693 15195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44687
I20260924 04:38:28.153995 15262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44687 every 8 connection(s)
I20260924 04:38:28.154726 15195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:38:28.158840 15263 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:38:28.160426 15263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Bootstrap starting.
I20260924 04:38:28.161042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 15195
I20260924 04:38:28.161475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36311
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44687
--webserver_port=41567
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:38:28.177235 15263 log.cc:824] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:28.187889 15263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:38:28.188239 15263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d: Bootstrap complete.
I20260924 04:38:28.192452 15263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:28.192899 15263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e63ac4690f4539b6ee74bb7b215c5d, State: Initialized, Role: FOLLOWER
I20260924 04:38:28.193404 15263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:28.193543 15263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:38:28.193614 15263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:38:28.193707 15263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:38:28.195485 15263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:28.195803 15263 leader_election.cc:302] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e63ac4690f4539b6ee74bb7b215c5d; no voters:
I20260924 04:38:28.196094 15263 leader_election.cc:288] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:38:28.196218 15267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:38:28.196466 15267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [term 3 LEADER]: Becoming Leader. State: Replica: 81e63ac4690f4539b6ee74bb7b215c5d, State: Running, Role: LEADER
I20260924 04:38:28.196756 15267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } }
I20260924 04:38:28.197252 15263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:28.198072 15268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } } }
I20260924 04:38:28.198197 15268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:28.198855 15269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e63ac4690f4539b6ee74bb7b215c5d. Latest consensus state: current_term: 3 leader_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e63ac4690f4539b6ee74bb7b215c5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44687 } } }
I20260924 04:38:28.198969 15269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e63ac4690f4539b6ee74bb7b215c5d [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:28.199246 15276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:38:28.204479 15276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:38:28.205166 15276 catalog_manager.cc:1259] Loaded cluster ID: 39079105aa1549f89c91c8a9653aa73f
I20260924 04:38:28.205230 15276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 04:38:28.206497 15276 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcbcd3be9ad google::LogMessage::Fail() at ??:0
@ 0x7fcbcd3c188a google::LogMessage::SendToLog() at ??:0
@ 0x7fcbcd3be4de google::LogMessage::Flush() at ??:0
@ 0x7fcbcd3bfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcbd7cfdfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fcbd7cfed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcbd7cfb579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fcbd7d3f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcbcf6103d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcbcf610c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fcbcf6123f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcbcf601512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcbd0c456db start_thread at ??:0
@ 0x7fcbcbebc61f clone at ??:0
W20260924 04:38:28.288810 15265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:28.289180 15265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:28.289278 15265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:28.293504 15265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:28.293576 15265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:28.293684 15265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:38:28.298921 15265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41567
--tserver_master_addrs=127.0.98.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:28.299927 15265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:28.300915 15265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:28.310349 15305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:28.310356 15306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:28.310431 15308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:28.311271 15265 server_base.cc:1034] running on GCE node
I20260924 04:38:28.311648 15265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:28.312279 15265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:28.313446 15265 hybrid_clock.cc:648] HybridClock initialized: now 1790224708313410 us; error 51 us; skew 500 ppm
Sep 24 04:38:28 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:28.316169 15265 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:38:28.317070 15265 webserver.cc:460] Webserver started at http://127.0.98.1:41567/ using document root <none> and password file <none>
I20260924 04:38:28.317626 15265 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:28.317687 15265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:28.321444 15265 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 04:38:28.323508 15315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:28.324733 15265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:28.324821 15265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "43080799469e4603848160cda44b6c3e"
format_stamp: "Formatted at 2026-09-24 04:38:25 on dist-test-slave-jlnt"
server_key: "f0c904134dbb25b1627c7f1da97d84ca"
server_key_iv: "a0e999f92dc52fb6632667d4ba2db5a7"
server_key_version: "encryptionkey@0"
I20260924 04:38:28.325413 15265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:28.355350 15265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:28.358592 15265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:28.358825 15265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:28.359396 15265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:28.360530 15322 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:38:28.365909 15265 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:38:28.365988 15265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:38:28.366060 15265 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:38:28.367614 15265 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:38:28.367683 15265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:38:28.367959 15322 tablet_bootstrap.cc:492] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Bootstrap starting.
I20260924 04:38:28.378196 15265 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36311
I20260924 04:38:28.378327 15429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36311 every 8 connection(s)
I20260924 04:38:28.379123 15265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:38:28.387985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 15265
I20260924 04:38:28.388406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32945
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44687
--webserver_port=43221
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:38:28.389792 15322 log.cc:824] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:28.398943 15322 tablet_bootstrap.cc:492] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:38:28.399367 15322 tablet_bootstrap.cc:492] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Bootstrap complete.
I20260924 04:38:28.400681 15322 ts_tablet_manager.cc:1397] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260924 04:38:28.403998 15322 raft_consensus.cc:348] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:28.404532 15322 raft_consensus.cc:729] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43080799469e4603848160cda44b6c3e, State: Initialized, Role: FOLLOWER
I20260924 04:38:28.405031 15322 consensus_queue.cc:260] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:28.406117 15322 ts_tablet_manager.cc:1428] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260924 04:38:28.458669 15432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:38:28.380968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:39377)
0924 04:38:28.381450 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:28.381500 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:28.382284 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:28.457864 (+ 75580us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":151,"threads_started":1}
W20260924 04:38:28.459179 15430 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44687 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44687: Client connection negotiation failed: client connection to 127.0.98.62:44687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:38:28.459945 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:28.459702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:60897)
0924 04:38:28.459826 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:28.459889 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 04:38:28.460692 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:28.460472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:46917)
0924 04:38:28.460596 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:28.460647 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:38:28.460925 15430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:38:28.502326 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:28.502573 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:28.502650 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:28.506656 15433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:28.506718 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:28.506800 15433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:38:28.512044 15433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43221
--tserver_master_addrs=127.0.98.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:28.513042 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:28.514048 15433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:28.523573 15441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:28.523691 15433 server_base.cc:1034] running on GCE node
W20260924 04:38:28.523634 15440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:28.523634 15443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:28.524351 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:28.524931 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:28.526090 15433 hybrid_clock.cc:648] HybridClock initialized: now 1790224708526067 us; error 41 us; skew 500 ppm
Sep 24 04:38:28 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:28.528946 15433 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:38:28.529959 15433 webserver.cc:460] Webserver started at http://127.0.98.2:43221/ using document root <none> and password file <none>
I20260924 04:38:28.530468 15433 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:28.530529 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:28.534271 15433 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:38:28.536270 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:28.537626 15433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:38:28.537731 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "bbd3cf416da84f9e9907898c53b976c3"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "9ca15f56b042bc8d5bab52f3caac3c88"
server_key_iv: "ae3bc381e23999bf5f567efaa4fcf8d6"
server_key_version: "encryptionkey@0"
I20260924 04:38:28.538260 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:28.569394 15433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:28.572480 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:28.572654 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:28.573253 15433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:28.574026 15433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:28.574071 15433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:28.574110 15433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:28.574126 15433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:28.582839 15433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32945
I20260924 04:38:28.582866 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32945 every 8 connection(s)
I20260924 04:38:28.583811 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:38:28.585127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 15433
I20260924 04:38:28.585529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39815
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44687
--webserver_port=39063
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:38:28.587599 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:28.586099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:47023)
0924 04:38:28.586783 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:28.586905 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":122,"threads_started":1}
W20260924 04:38:28.588032 15564 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44687 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44687: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
I20260924 04:38:28.588680 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:28.588425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:60287)
0924 04:38:28.588574 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:28.588624 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 04:38:28.589365 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:28.589175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:37571)
0924 04:38:28.589277 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:28.589317 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:38:28.589673 15564 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:38:28.692121 15567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:28.692354 15567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:28.692427 15567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:28.696349 15567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:28.696408 15567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:28.696461 15567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:38:28.701598 15567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39063
--tserver_master_addrs=127.0.98.62:44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:28.702596 15567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:28.703593 15567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:28.713172 15573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:28.713217 15572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:28.713153 15575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:28.713368 15567 server_base.cc:1034] running on GCE node
I20260924 04:38:28.713815 15567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:28.714448 15567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:28.715638 15567 hybrid_clock.cc:648] HybridClock initialized: now 1790224708715620 us; error 27 us; skew 500 ppm
Sep 24 04:38:28 dist-test-slave-jlnt krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:28.718483 15567 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:38:28.719340 15567 webserver.cc:460] Webserver started at http://127.0.98.3:39063/ using document root <none> and password file <none>
I20260924 04:38:28.719915 15567 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:28.720007 15567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:28.723773 15567 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:28.725834 15582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:28.727033 15567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:28.727165 15567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "4a9c944515244f95a74840be8b1709e1"
format_stamp: "Formatted at 2026-09-24 04:38:26 on dist-test-slave-jlnt"
server_key: "36e2be089fd855e4a49068c08853fa73"
server_key_iv: "f5ce0c0c85e34f2ef296f723c0856cc8"
server_key_version: "encryptionkey@0"
I20260924 04:38:28.727684 15567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:28.745299 15567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:28.748413 15567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:28.748605 15567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:28.749169 15567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:28.749957 15567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:28.750027 15567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:28.750090 15567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:28.750140 15567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:28.759087 15567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39815
I20260924 04:38:28.759111 15695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39815 every 8 connection(s)
I20260924 04:38:28.760035 15567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:38:28.761700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 15567
I20260924 04:38:28.763720 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:28.762202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:34277)
0924 04:38:28.762966 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:28.763063 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":155,"threads_started":1}
W20260924 04:38:28.764150 15696 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44687 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44687: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
I20260924 04:38:28.764657 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:28.764467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:42333)
0924 04:38:28.764562 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:28.764605 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:38:28.765257 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:28.765045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:38223)
0924 04:38:28.765129 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:28.765211 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:38:28.765425 15696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:38:29.461948 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:29.461515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:52457)
0924 04:38:29.461774 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:29.461825 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260924 04:38:29.590601 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:29.590203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:42129)
0924 04:38:29.590448 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:29.590497 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":65,"threads_started":1}
I20260924 04:38:29.766327 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:29.765944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:54509)
0924 04:38:29.766169 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:29.766221 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":67,"threads_started":1}
I20260924 04:38:30.115645 15703 raft_consensus.cc:482] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:30.115849 15703 raft_consensus.cc:504] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:30.116832 15703 leader_election.cc:288] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815), bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945)
Sep 24 04:38:30 dist-test-slave-jlnt krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 04:38:30 dist-test-slave-jlnt krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 04:38:30.134161 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:30.117267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46307 (local address 127.0.98.2:32945)
0924 04:38:30.117688 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:30.117705 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:30.117754 (+ 49us) server_negotiation.cc:407] Connection header received
0924 04:38:30.118642 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:30.118651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:30.119117 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:30.119470 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:30.121763 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:30.123211 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:30.125479 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:30.126085 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:30.126268 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:30.129363 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:30.129368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:30.129379 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:30.129408 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:30.131438 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:30.132056 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:30.132066 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:30.132070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:30.132181 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:30.132602 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:30.132605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:30.132607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:30.132998 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:30.133189 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:30.133712 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 04:38:30.133973 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":93,"threads_started":1}
I20260924 04:38:30.134186 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:30.117075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32945 (local address 127.0.98.1:46307)
0924 04:38:30.117477 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:30.117495 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:30.117622 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:30.119607 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:30.119615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:30.120281 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:30.121371 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:30.121401 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:30.123403 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:30.123406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:30.125302 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:30.125315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:30.126231 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:30.126235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:30.126450 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:30.127269 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:30.127282 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:30.129123 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:30.131562 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:30.131568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:30.131582 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:30.131934 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:30.132298 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:30.132303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:30.132305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:30.132476 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:30.133151 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:30.133157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:30.133404 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 04:38:30.133582 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 04:38:30.133959 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":512,"spinlock_wait_cycles":5760,"thread_start_us":115,"threads_started":1}
I20260924 04:38:30.134290 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:30.117075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39815 (local address 127.0.98.1:47325)
0924 04:38:30.117577 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:30.117590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:30.117659 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:30.119607 (+ 1948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:30.119615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:30.120281 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:30.121371 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:30.121398 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:30.123395 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:30.123399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:30.125302 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:30.125315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:30.126418 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:30.126422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:30.126505 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:30.127193 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:30.127209 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:30.128874 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:30.131616 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:30.131620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:30.131622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:30.131975 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:30.132353 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:30.132356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:30.132358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:30.132476 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:30.133161 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:30.133165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:30.133427 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 04:38:30.133583 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 04:38:30.134051 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":565,"spinlock_wait_cycles":55936,"thread_start_us":59,"threads_started":1}
I20260924 04:38:30.134665 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:30.117267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47325 (local address 127.0.98.3:39815)
0924 04:38:30.117689 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:30.117722 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:30.117754 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:38:30.118642 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:30.118651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:30.119117 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:30.119470 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:30.121593 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:30.123207 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:30.125479 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:30.126281 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:30.126450 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:30.129218 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:30.129226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:30.129266 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:30.129310 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:30.131466 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:30.132104 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:30.132110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:30.132116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:30.132242 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:30.132571 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:30.132575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:30.132577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:30.133006 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:30.133197 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:30.133737 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 04:38:30.134051 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":118,"threads_started":1}
W20260924 04:38:30.136755 15318 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
W20260924 04:38:30.136971 15317 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
I20260924 04:38:30.137085 15317 leader_election.cc:302] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43080799469e4603848160cda44b6c3e; no voters: 4a9c944515244f95a74840be8b1709e1, bbd3cf416da84f9e9907898c53b976c3
I20260924 04:38:30.137406 15703 raft_consensus.cc:2726] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:38:30.462960 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:30.462674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:58995)
0924 04:38:30.462806 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:30.462872 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:38:30.591838 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:30.591378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:56137)
0924 04:38:30.591640 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:30.591693 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260924 04:38:30.767803 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:30.767457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:55895)
0924 04:38:30.767670 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:30.767721 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260924 04:38:31.464102 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:31.463654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:55569)
0924 04:38:31.463938 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:31.463990 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
I20260924 04:38:31.592938 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:31.592508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:46275)
0924 04:38:31.592739 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:31.592797 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260924 04:38:31.768886 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:31.768424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:49501)
0924 04:38:31.768692 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:31.768750 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
I20260924 04:38:32.380545 15713 raft_consensus.cc:482] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:32.380658 15713 raft_consensus.cc:504] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:32.381161 15713 leader_election.cc:288] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815), bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945)
W20260924 04:38:32.381939 15317 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
W20260924 04:38:32.382045 15318 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
I20260924 04:38:32.382164 15318 leader_election.cc:302] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43080799469e4603848160cda44b6c3e; no voters: 4a9c944515244f95a74840be8b1709e1, bbd3cf416da84f9e9907898c53b976c3
I20260924 04:38:32.382333 15713 raft_consensus.cc:2726] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:38:32.465525 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:32.464918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:48355)
0924 04:38:32.465366 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:32.465428 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":303,"threads_started":1}
I20260924 04:38:32.594039 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:32.593681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:54119)
0924 04:38:32.593901 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:32.593950 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260924 04:38:32.770035 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:32.769672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:49031)
0924 04:38:32.769901 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:32.769952 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260924 04:38:33.466809 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:33.466359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:57683)
0924 04:38:33.466659 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:33.466715 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":72,"threads_started":1}
I20260924 04:38:33.595084 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:33.594693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:45249)
0924 04:38:33.594944 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:33.594997 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260924 04:38:33.771128 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:33.770709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:40485)
0924 04:38:33.770994 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:33.771045 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260924 04:38:34.467999 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:34.467603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:45491)
0924 04:38:34.467849 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:34.467905 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260924 04:38:34.514156 15721 raft_consensus.cc:482] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:34.514312 15721 raft_consensus.cc:504] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:34.514776 15721 leader_election.cc:288] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815), bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945)
W20260924 04:38:34.515560 15318 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
W20260924 04:38:34.515661 15317 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
I20260924 04:38:34.515806 15317 leader_election.cc:302] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43080799469e4603848160cda44b6c3e; no voters: 4a9c944515244f95a74840be8b1709e1, bbd3cf416da84f9e9907898c53b976c3
I20260924 04:38:34.515980 15721 raft_consensus.cc:2726] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:38:34.596119 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:34.595748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:32933)
0924 04:38:34.595976 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:34.596030 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260924 04:38:34.772193 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:34.771827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:55165)
0924 04:38:34.772061 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:34.772109 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260924 04:38:35.469336 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:35.468787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:57391)
0924 04:38:35.469075 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:35.469201 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260924 04:38:35.597352 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:35.596865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:49089)
0924 04:38:35.597193 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:35.597256 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":91,"threads_started":1}
I20260924 04:38:35.773386 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:35.772925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:34039)
0924 04:38:35.773242 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:35.773295 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":66,"threads_started":1}
I20260924 04:38:36.470455 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:36.470020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:60867)
0924 04:38:36.470304 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:36.470359 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260924 04:38:36.598425 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:36.598023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:49393)
0924 04:38:36.598287 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:36.598340 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260924 04:38:36.774461 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:36.774072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:49991)
0924 04:38:36.774309 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:36.774360 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260924 04:38:37.471745 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:37.471323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:34139)
0924 04:38:37.471579 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:37.471638 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
I20260924 04:38:37.599552 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:37.599120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:41435)
0924 04:38:37.599412 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:37.599464 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260924 04:38:37.775529 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:37.775168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:54853)
0924 04:38:37.775396 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:37.775446 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260924 04:38:38.473042 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:38.472648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:60401)
0924 04:38:38.472895 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:38.472949 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260924 04:38:38.600744 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:38.600298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:47681)
0924 04:38:38.600566 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:38.600618 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260924 04:38:38.653306 15735 raft_consensus.cc:482] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:38.653461 15735 raft_consensus.cc:504] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:38.653894 15735 leader_election.cc:288] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815), bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945)
W20260924 04:38:38.654637 15317 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
W20260924 04:38:38.654742 15318 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
I20260924 04:38:38.654868 15318 leader_election.cc:302] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43080799469e4603848160cda44b6c3e; no voters: 4a9c944515244f95a74840be8b1709e1, bbd3cf416da84f9e9907898c53b976c3
I20260924 04:38:38.655022 15735 raft_consensus.cc:2726] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:38:38.776626 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:38.776240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:55217)
0924 04:38:38.776484 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:38.776535 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260924 04:38:39.474275 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:39.473900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:35973)
0924 04:38:39.474127 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:39.474178 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260924 04:38:39.601962 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:39.601543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:49533)
0924 04:38:39.601792 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:39.601845 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260924 04:38:39.777830 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:39.777413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:43225)
0924 04:38:39.777678 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:39.777737 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260924 04:38:40.475461 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:40.475023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:44583)
0924 04:38:40.475303 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:40.475361 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
I20260924 04:38:40.603096 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:40.602722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:46899)
0924 04:38:40.602955 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:40.603006 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260924 04:38:40.778936 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:40.778535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:52155)
0924 04:38:40.778790 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:40.778843 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260924 04:38:41.369385 15743 raft_consensus.cc:482] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:41.369539 15743 raft_consensus.cc:504] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a9c944515244f95a74840be8b1709e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39815 } } peers { permanent_uuid: "bbd3cf416da84f9e9907898c53b976c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32945 } } peers { permanent_uuid: "43080799469e4603848160cda44b6c3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36311 } }
I20260924 04:38:41.370012 15743 leader_election.cc:288] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815), bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945)
W20260924 04:38:41.370744 15317 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a9c944515244f95a74840be8b1709e1 (127.0.98.3:39815): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
W20260924 04:38:41.370898 15318 leader_election.cc:341] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbd3cf416da84f9e9907898c53b976c3 (127.0.98.2:32945): Not found: Tablet not found: 9c5a2b77dccf47ab8726f08450ff471d
I20260924 04:38:41.370981 15318 leader_election.cc:302] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43080799469e4603848160cda44b6c3e; no voters: 4a9c944515244f95a74840be8b1709e1, bbd3cf416da84f9e9907898c53b976c3
I20260924 04:38:41.371130 15743 raft_consensus.cc:2726] T 9c5a2b77dccf47ab8726f08450ff471d P 43080799469e4603848160cda44b6c3e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:38:41.476665 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:41.476243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:59905)
0924 04:38:41.476511 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:41.476569 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260924 04:38:41.604337 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:41.603922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:46977)
0924 04:38:41.604184 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:41.604240 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260924 04:38:41.780099 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:41.779691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:35915)
0924 04:38:41.779949 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:41.780008 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260924 04:38:42.477898 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:42.477501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:38193)
0924 04:38:42.477741 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:42.477803 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260924 04:38:42.605546 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:42.605109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:49163)
0924 04:38:42.605398 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:42.605454 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":119,"threads_started":1}
I20260924 04:38:42.781239 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:42.780817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.3:33163)
0924 04:38:42.781048 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:42.781098 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260924 04:38:43.479374 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:43.478964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.1:51293)
0924 04:38:43.479227 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:43.479280 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260924 04:38:43.606761 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:43.606387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44687 (local address 127.0.98.2:44017)
0924 04:38:43.606618 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:43.606670 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260924 04:38:43.762596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 15265
I20260924 04:38:43.768096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 15433
I20260924 04:38:43.773524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 15567
2026-09-24T04:38:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19333 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:38:43 dist-test-slave-jlnt krb5kdc[15754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:38:43 dist-test-slave-jlnt krb5kdc[15754](info): set up 2 sockets
Sep 24 04:38:43 dist-test-slave-jlnt krb5kdc[15754](info): commencing operation
krb5kdc: starting...
W20260924 04:38:44.809312 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:38:44 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:38:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:38:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:38:44.928625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36685
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:45.034605 15767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:45.034847 15767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:45.034885 15767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:45.038713 15767 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:38:45.038751 15767 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:38:45.038769 15767 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:38:45.038782 15767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:38:45.038800 15767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:45.038817 15767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:45.038833 15767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:38:45.038852 15767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:38:45.043788 15767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:45.044649 15767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:45.045562 15767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:45.053875 15775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:45.053892 15773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:45.053918 15767 server_base.cc:1034] running on GCE node
W20260924 04:38:45.053876 15772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:45.054692 15767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:45.055903 15767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:45.057109 15767 hybrid_clock.cc:648] HybridClock initialized: now 1790224725057095 us; error 35 us; skew 500 ppm
Sep 24 04:38:45 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:45.059859 15767 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:38:45.060688 15767 webserver.cc:460] Webserver started at http://127.0.98.62:39137/ using document root <none> and password file <none>
I20260924 04:38:45.061349 15767 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:45.061435 15767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:45.061663 15767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:45.063589 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "4a66bd231e604fada64bec2070732cc3"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "34d867946af5e3d3f66005d78b853ebb"
server_key_iv: "62212f89f41dbf6f0f856cea44ac29ad"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.064172 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a66bd231e604fada64bec2070732cc3"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "34d867946af5e3d3f66005d78b853ebb"
server_key_iv: "62212f89f41dbf6f0f856cea44ac29ad"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.067939 15767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 04:38:45.070448 15782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.071631 15767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:38:45.071768 15767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "4a66bd231e604fada64bec2070732cc3"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "34d867946af5e3d3f66005d78b853ebb"
server_key_iv: "62212f89f41dbf6f0f856cea44ac29ad"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.071907 15767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:45.087492 15767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:45.090454 15767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:45.090662 15767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:45.097353 15767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36685
I20260924 04:38:45.097620 15834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36685 every 8 connection(s)
I20260924 04:38:45.098295 15767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:38:45.100925 15835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:45.104571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 15767
I20260924 04:38:45.104698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:38:45.104961 392 external_mini_cluster.cc:1428] Setting key 1ef24dbe40dfc9f9dc4a2ffda1af1491
I20260924 04:38:45.106644 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Bootstrap starting.
I20260924 04:38:45.109210 15835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:45.110067 15835 log.cc:824] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:45.111778 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: No bootstrap required, opened a new log
Sep 24 04:38:45 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:45.116708 15835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:38:45.116981 15835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:45.117067 15835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a66bd231e604fada64bec2070732cc3, State: Initialized, Role: FOLLOWER
I20260924 04:38:45.117594 15835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:38:45.117753 15835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:38:45.117841 15835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:38:45.117944 15835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:45.119113 15835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:38:45.119464 15835 leader_election.cc:302] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a66bd231e604fada64bec2070732cc3; no voters:
I20260924 04:38:45.119841 15835 leader_election.cc:288] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:38:45.120167 15841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:38:45.120393 15841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 1 LEADER]: Becoming Leader. State: Replica: 4a66bd231e604fada64bec2070732cc3, State: Running, Role: LEADER
I20260924 04:38:45.120728 15841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:38:45.120865 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.106354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.0.98.62:36685)
0924 04:38:45.106759 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.106781 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.106823 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:38:45.107615 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.107638 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.108069 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.108416 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:45.109114 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.110583 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.111443 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.111937 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.112113 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.115240 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:45.115249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:45.115270 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:45.115308 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:45.117417 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.117926 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.117933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.117938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.118070 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.118409 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.118413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.118415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.118908 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:45.119132 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.119439 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.119930 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260924 04:38:45.121925 15841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a66bd231e604fada64bec2070732cc3. Latest consensus state: current_term: 1 leader_uuid: "4a66bd231e604fada64bec2070732cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } } }
I20260924 04:38:45.121093 15835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:38:45.122099 15841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:45.124042 15854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:38:45.126767 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a66bd231e604fada64bec2070732cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } } }
I20260924 04:38:45.126883 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:38:45.127734 15854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:38:45.135300 15854 catalog_manager.cc:1347] Generated new cluster ID: 303a3da5cf544fe7b821f470c9f64ab0
I20260924 04:38:45.135401 15854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:38:45.152199 15854 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:38:45.153179 15854 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:38:45.159600 15854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Generated new TSK 0
I20260924 04:38:45.160262 15854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:38:45.194388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36685
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:45.301496 15861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:45.301744 15861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:45.301826 15861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:45.305799 15861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:45.305851 15861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:45.305931 15861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:38:45.310973 15861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:45.311992 15861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:45.312989 15861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:45.322183 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:45.322170 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:45.322384 15861 server_base.cc:1034] running on GCE node
W20260924 04:38:45.322779 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:45.323405 15861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:45.324314 15861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:45.325501 15861 hybrid_clock.cc:648] HybridClock initialized: now 1790224725325470 us; error 46 us; skew 500 ppm
Sep 24 04:38:45 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:45.328549 15861 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:38:45.329434 15861 webserver.cc:460] Webserver started at http://127.0.98.1:35441/ using document root <none> and password file <none>
I20260924 04:38:45.330092 15861 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:45.330154 15861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:45.330312 15861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:45.332192 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "b6cf48e1833b403bb5546aebf0181f1c"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "ecc859cb04aa72cd9105b41bf79dfbee"
server_key_iv: "78f0ec6e5ef442477c295ba268f48c01"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.332696 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b6cf48e1833b403bb5546aebf0181f1c"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "ecc859cb04aa72cd9105b41bf79dfbee"
server_key_iv: "78f0ec6e5ef442477c295ba268f48c01"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.336382 15861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:38:45.338845 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.340150 15861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:38:45.340281 15861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "b6cf48e1833b403bb5546aebf0181f1c"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "ecc859cb04aa72cd9105b41bf79dfbee"
server_key_iv: "78f0ec6e5ef442477c295ba268f48c01"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.340399 15861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:45.351702 15861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:45.354916 15861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:45.355111 15861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:45.355683 15861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:45.356477 15861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:45.356521 15861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.356585 15861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:45.356616 15861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.365434 15861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40195
I20260924 04:38:45.365464 15989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40195 every 8 connection(s)
I20260924 04:38:45.366286 15861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:38:45.370297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 15861
I20260924 04:38:45.370432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:38:45.370779 392 external_mini_cluster.cc:1428] Setting key c6e273e12e8058e7bb2f9e31ddb7d1c4
Sep 24 04:38:45 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:38:45.384243 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.368090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50721 (local address 127.0.98.62:36685)
0924 04:38:45.368239 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.368244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.369108 (+ 864us) server_negotiation.cc:407] Connection header received
0924 04:38:45.370430 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.370433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.370489 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.370574 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:45.372056 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.372668 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.374074 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.374481 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.374613 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.379274 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:45.379278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:45.379280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:45.379339 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:45.381434 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.382072 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.382075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.382077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.382180 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.382845 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.382848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.382850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.383105 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:45.383215 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.383820 (+ 605us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.383965 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:38:45.385628 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.368527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:50721)
0924 04:38:45.368961 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:45.369002 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:45.370216 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:45.370801 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:45.370812 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:45.371290 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:45.371901 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.371921 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.372819 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.372822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.373955 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.373964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.374684 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.374688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.375853 (+ 1165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.376498 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:45.376517 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:45.379060 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:45.381547 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:45.381554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:45.381573 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:45.381947 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:45.382288 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:45.382291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:45.382293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:45.382730 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:45.383224 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:45.383233 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:45.383531 (+ 298us) client_negotiation.cc:769] Sending connection context
0924 04:38:45.383773 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 04:38:45.384315 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260924 04:38:45.386965 15990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36685
I20260924 04:38:45.387223 15990 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:45.387717 15990 heartbeater.cc:511] Master 127.0.98.62:36685 requested a full tablet report, sending...
I20260924 04:38:45.389374 15799 ts_manager.cc:194] Registered new tserver with Master: b6cf48e1833b403bb5546aebf0181f1c (127.0.98.1:40195)
I20260924 04:38:45.391104 15799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:38:45.404139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36685
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:45.519336 15995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:45.519590 15995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:45.519681 15995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:45.523896 15995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:45.523972 15995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:45.524055 15995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:38:45.529258 15995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:45.530268 15995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:45.531234 15995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:45.540509 16000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:45.540530 16003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:45.540519 16001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:45.540843 15995 server_base.cc:1034] running on GCE node
I20260924 04:38:45.541307 15995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:45.541913 15995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:45.543069 15995 hybrid_clock.cc:648] HybridClock initialized: now 1790224725543055 us; error 27 us; skew 500 ppm
Sep 24 04:38:45 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:45.545900 15995 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:38:45.546671 15995 webserver.cc:460] Webserver started at http://127.0.98.2:32873/ using document root <none> and password file <none>
I20260924 04:38:45.547175 15995 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:45.547235 15995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:45.547394 15995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:45.549392 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "9579d1e3079a4ad29441c5c966335347"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "b8a7102acb85cd698efe26d8d37f4900"
server_key_iv: "50e0520d468834fd1adaef6d14d4ed97"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.549918 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9579d1e3079a4ad29441c5c966335347"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "b8a7102acb85cd698efe26d8d37f4900"
server_key_iv: "50e0520d468834fd1adaef6d14d4ed97"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.553489 15995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:38:45.555758 16010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.556994 15995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:38:45.557140 15995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "9579d1e3079a4ad29441c5c966335347"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "b8a7102acb85cd698efe26d8d37f4900"
server_key_iv: "50e0520d468834fd1adaef6d14d4ed97"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.557286 15995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:45.585538 15995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:45.588382 15995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:45.588585 15995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:45.589193 15995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:45.590080 15995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:45.590154 15995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.590224 15995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:45.590274 15995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.599021 15995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34097
I20260924 04:38:45.599040 16123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34097 every 8 connection(s)
I20260924 04:38:45.599874 15995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:38:45.601022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 15995
I20260924 04:38:45.601130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:38:45.601445 392 external_mini_cluster.cc:1428] Setting key 928d3a00e1afe743a4d40cf2f955632a
Sep 24 04:38:45 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:38:45.615046 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.601809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54569 (local address 127.0.98.62:36685)
0924 04:38:45.601948 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.601952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.602795 (+ 843us) server_negotiation.cc:407] Connection header received
0924 04:38:45.604056 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.604062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.604125 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.604222 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:45.605826 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.606344 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.608191 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.608415 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.608493 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.611394 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:45.611397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:45.611398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:45.611423 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:45.612870 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.613432 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.613435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.613436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.613512 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.613857 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.613860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.613861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.614114 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:45.614244 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.614746 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.614873 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:38:45.615891 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.602205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:54569)
0924 04:38:45.602640 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:45.602691 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:45.603854 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:45.604434 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:45.604444 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:45.604954 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:45.605675 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.605697 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.606470 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.606477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.608047 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.608062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.608571 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.608578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.608793 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.609468 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:45.609488 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:45.611210 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:45.612976 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:45.612980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:45.612992 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:45.613345 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:45.613594 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:45.613596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:45.613597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:45.613723 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:45.614234 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:45.614241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:45.614500 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 04:38:45.614687 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 04:38:45.615018 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
I20260924 04:38:45.617239 16124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36685
I20260924 04:38:45.617491 16124 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:45.618044 16124 heartbeater.cc:511] Master 127.0.98.62:36685 requested a full tablet report, sending...
I20260924 04:38:45.619091 15799 ts_manager.cc:194] Registered new tserver with Master: 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097)
I20260924 04:38:45.619673 15799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:38:45.632366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36685
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:38:45.740653 16129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:38:45.740939 16129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:38:45.741026 16129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:38:45.745337 16129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:38:45.745415 16129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:38:45.745508 16129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:38:45.750775 16129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:38:45.751818 16129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:38:45.752833 16129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:38:45.762498 16134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:45.762485 16135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:38:45.762486 16137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:38:45.762738 16129 server_base.cc:1034] running on GCE node
I20260924 04:38:45.763195 16129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:38:45.763794 16129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:38:45.764952 16129 hybrid_clock.cc:648] HybridClock initialized: now 1790224725764938 us; error 26 us; skew 500 ppm
Sep 24 04:38:45 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:38:45.767782 16129 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:38:45.768574 16129 webserver.cc:460] Webserver started at http://127.0.98.3:46315/ using document root <none> and password file <none>
I20260924 04:38:45.769106 16129 fs_manager.cc:359] Metadata directory not provided
I20260924 04:38:45.769251 16129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:38:45.769476 16129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:38:45.771440 16129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "dfac52afbf744ee0969a49bd9dde53b8"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "5b7cf01153c577ba9356e8a47de7738e"
server_key_iv: "8caec76807069b445f7a5dafaadee133"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.771984 16129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dfac52afbf744ee0969a49bd9dde53b8"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "5b7cf01153c577ba9356e8a47de7738e"
server_key_iv: "8caec76807069b445f7a5dafaadee133"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.775606 16129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:38:45.778180 16144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.779417 16129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:38:45.779560 16129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "dfac52afbf744ee0969a49bd9dde53b8"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "5b7cf01153c577ba9356e8a47de7738e"
server_key_iv: "8caec76807069b445f7a5dafaadee133"
server_key_version: "encryptionkey@0"
I20260924 04:38:45.779675 16129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:38:45.807992 16129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:38:45.811376 16129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:38:45.811523 16129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:38:45.812033 16129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:38:45.812778 16129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:38:45.812824 16129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.812862 16129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:38:45.812877 16129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:38:45.821413 16129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38973
I20260924 04:38:45.821484 16257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38973 every 8 connection(s)
I20260924 04:38:45.822294 16129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:38:45.828570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 16129
I20260924 04:38:45.828701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:38:45.828999 392 external_mini_cluster.cc:1428] Setting key 7156da3b79ef5d90b97cc28e57cd59a4
Sep 24 04:38:45 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:38:45.836131 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.824210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50503 (local address 127.0.98.62:36685)
0924 04:38:45.824366 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.824369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.825292 (+ 923us) server_negotiation.cc:407] Connection header received
0924 04:38:45.826713 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.826716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.826775 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.826861 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:45.828023 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.828595 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.830207 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.830499 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.830580 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.833200 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:45.833204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:45.833206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:45.833229 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:45.834452 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.834893 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.834895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.834897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.834970 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.835249 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.835251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.835252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.835428 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:45.835500 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.835920 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.836014 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:38:45.836915 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.824600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:50503)
0924 04:38:45.825088 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:45.825131 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:45.826505 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:45.827041 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:45.827047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:45.827398 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:38:45.827882 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.827895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.828795 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.828799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.830070 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.830082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.830638 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.830643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.830832 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.831522 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:38:45.831537 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:38:45.833009 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:38:45.834551 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:45.834555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:45.834567 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:45.834808 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:45.835047 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:38:45.835049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:38:45.835050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:38:45.835175 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:38:45.835510 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:38:45.835515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:38:45.835731 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 04:38:45.835876 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 04:38:45.836116 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":97,"threads_started":1}
I20260924 04:38:45.838047 16258 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36685
I20260924 04:38:45.838263 16258 heartbeater.cc:464] Registering TS with master...
I20260924 04:38:45.838724 16258 heartbeater.cc:511] Master 127.0.98.62:36685 requested a full tablet report, sending...
I20260924 04:38:45.839694 15799 ts_manager.cc:194] Registered new tserver with Master: dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973)
I20260924 04:38:45.840230 15799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50503
I20260924 04:38:45.842803 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:38:45.851562 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.844727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.0.98.62:36685)
0924 04:38:45.844854 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.844857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.845003 (+ 146us) server_negotiation.cc:407] Connection header received
0924 04:38:45.845100 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.845102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.845170 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.845259 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:38:45.845942 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.846429 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.847218 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.847423 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.847498 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.848771 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:38:45.848774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:38:45.848775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:38:45.848794 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:38:45.850055 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.850409 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.850411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.850412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.850479 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:38:45.850736 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:38:45.850741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:38:45.850743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:38:45.851079 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:38:45.851192 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.851367 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.851454 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:38:45.854135 15799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:38:45.857045 15799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:38:45.870244 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.864946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34097 (local address 127.0.0.1:38010)
0924 04:38:45.865311 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:45.865367 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:45.865507 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:45.866149 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:45.866155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:45.866179 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:45.866303 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.866307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.867727 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.867733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.868644 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.868654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.869709 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.869713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.869823 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.869858 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:38:45.869976 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:38:45.870098 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
I20260924 04:38:45.870254 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.865163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.0.98.2:34097)
0924 04:38:45.865640 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.865652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.865674 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:38:45.865750 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.865755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.865973 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.866475 (+ 502us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:45.866646 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.867596 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.868783 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.869574 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.869713 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.869866 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.870005 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.870098 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":131,"threads_started":1}
I20260924 04:38:45.871821 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.865744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40195 (local address 127.0.0.1:35348)
0924 04:38:45.865993 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:45.866005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:45.866094 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:45.867552 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:45.867554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:45.867573 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:45.867678 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.867684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.869033 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.869036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.870283 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.870294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.871402 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.871405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.871506 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.871530 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:38:45.871609 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:38:45.871676 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":41,"thread_start_us":115,"threads_started":1}
I20260924 04:38:45.871922 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.866060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.0.98.1:40195)
0924 04:38:45.866948 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.866961 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.866978 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:38:45.867050 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.867054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.867202 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.867363 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:45.867810 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.868895 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.870432 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.871253 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.871385 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.871536 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.871658 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.871755 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":81,"threads_started":1}
I20260924 04:38:45.872131 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.866062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38973 (local address 127.0.0.1:33254)
0924 04:38:45.866624 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:45.866635 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:45.866734 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:45.867344 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:45.867347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:45.867365 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:45.867468 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.867474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.868916 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.868922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.870283 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.870294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.871720 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.871723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.871817 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.871841 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:38:45.871912 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:38:45.871997 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":43,"thread_start_us":84,"threads_started":1}
I20260924 04:38:45.872530 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.866493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.0.98.3:38973)
0924 04:38:45.866872 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.866884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.866907 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:38:45.866996 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.867003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.867232 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.867363 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:45.867557 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.868756 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.870636 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.871422 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.871514 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.871657 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.872304 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.872398 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":167,"threads_started":1}
I20260924 04:38:45.874303 16058 tablet_service.cc:1467] Processing CreateTablet for tablet 709fdd6e68714c09983375ccf1e9b268 (DEFAULT_TABLE table=test-table [id=cf453024494845c382157b2dac74abed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:45.874956 15924 tablet_service.cc:1467] Processing CreateTablet for tablet 709fdd6e68714c09983375ccf1e9b268 (DEFAULT_TABLE table=test-table [id=cf453024494845c382157b2dac74abed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:45.875550 16058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709fdd6e68714c09983375ccf1e9b268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:45.875990 15924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709fdd6e68714c09983375ccf1e9b268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:45.882037 16275 tablet_bootstrap.cc:492] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Bootstrap starting.
I20260924 04:38:45.883266 16276 tablet_bootstrap.cc:492] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Bootstrap starting.
I20260924 04:38:45.883375 16192 tablet_service.cc:1467] Processing CreateTablet for tablet 709fdd6e68714c09983375ccf1e9b268 (DEFAULT_TABLE table=test-table [id=cf453024494845c382157b2dac74abed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:38:45.884438 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709fdd6e68714c09983375ccf1e9b268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:38:45.885491 16276 tablet_bootstrap.cc:654] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:45.885927 16275 tablet_bootstrap.cc:654] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:45.886813 16275 log.cc:824] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:45.886813 16276 log.cc:824] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:45.888885 16275 tablet_bootstrap.cc:492] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: No bootstrap required, opened a new log
I20260924 04:38:45.888882 16276 tablet_bootstrap.cc:492] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: No bootstrap required, opened a new log
I20260924 04:38:45.889083 16275 ts_tablet_manager.cc:1397] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:38:45.889083 16276 ts_tablet_manager.cc:1397] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:38:45.891114 16279 tablet_bootstrap.cc:492] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: Bootstrap starting.
I20260924 04:38:45.893116 16275 raft_consensus.cc:348] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:38:45.893116 16276 raft_consensus.cc:348] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:38:45.893436 16276 raft_consensus.cc:374] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:45.893436 16275 raft_consensus.cc:374] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:45.893505 16275 raft_consensus.cc:729] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9579d1e3079a4ad29441c5c966335347, State: Initialized, Role: FOLLOWER
I20260924 04:38:45.893505 16276 raft_consensus.cc:729] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cf48e1833b403bb5546aebf0181f1c, State: Initialized, Role: FOLLOWER
I20260924 04:38:45.893637 16279 tablet_bootstrap.cc:654] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: Neither blocks nor log segments found. Creating new log.
I20260924 04:38:45.894021 16276 consensus_queue.cc:260] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:38:45.894021 16275 consensus_queue.cc:260] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:38:45.894735 16124 heartbeater.cc:503] Master 127.0.98.62:36685 was elected leader, sending a full tablet report...
I20260924 04:38:45.895059 16275 ts_tablet_manager.cc:1428] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260924 04:38:45.895059 16276 ts_tablet_manager.cc:1428] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260924 04:38:45.895398 15990 heartbeater.cc:503] Master 127.0.98.62:36685 was elected leader, sending a full tablet report...
I20260924 04:38:45.895782 16279 log.cc:824] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:38:45.898104 16279 tablet_bootstrap.cc:492] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: No bootstrap required, opened a new log
I20260924 04:38:45.898381 16279 ts_tablet_manager.cc:1397] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:38:45.901712 16279 raft_consensus.cc:348] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:38:45.901949 16279 raft_consensus.cc:374] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:38:45.902009 16279 raft_consensus.cc:729] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfac52afbf744ee0969a49bd9dde53b8, State: Initialized, Role: FOLLOWER
I20260924 04:38:45.902479 16279 consensus_queue.cc:260] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:38:45.903156 16258 heartbeater.cc:503] Master 127.0.98.62:36685 was elected leader, sending a full tablet report...
I20260924 04:38:45.903510 16279 ts_tablet_manager.cc:1428] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:38:45.942823 16283 raft_consensus.cc:482] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:38:45.943079 16283 raft_consensus.cc:504] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:38:45.944391 16283 leader_election.cc:288] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097), b6cf48e1833b403bb5546aebf0181f1c (127.0.98.1:40195)
I20260924 04:38:45.949071 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.944623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34097 (local address 127.0.98.3:32953)
0924 04:38:45.944762 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:45.944781 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:45.944886 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:45.945314 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:45.945318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:45.945336 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:45.945479 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.945489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.946506 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.946509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.947717 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.947727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.948628 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.948631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.948709 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.948760 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 04:38:45.948839 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:38:45.948914 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 04:38:45.949088 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.944729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32953 (local address 127.0.98.2:34097)
0924 04:38:45.944869 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.944874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.944890 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:38:45.945022 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.945027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.945081 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.945183 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:45.945601 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.946399 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.947857 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.948511 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.948626 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.948693 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.948864 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.948937 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:38:45.949247 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.945107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36283 (local address 127.0.98.1:40195)
0924 04:38:45.945314 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.945319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.945438 (+ 119us) server_negotiation.cc:407] Connection header received
0924 04:38:45.945581 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.945585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.945647 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.945749 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:38:45.946004 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.946674 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.948038 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.948668 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.948761 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.948819 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.949033 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.949108 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260924 04:38:45.949612 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.944987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40195 (local address 127.0.98.3:36283)
0924 04:38:45.945356 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:45.945369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:38:45.945460 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:38:45.945759 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:38:45.945762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:38:45.945779 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:38:45.945892 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.945897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.946781 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.946784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.947919 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:38:45.947926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.948777 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.948781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:38:45.948913 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.948931 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:38:45.949000 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:38:45.949051 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":143,"threads_started":1}
I20260924 04:38:45.949862 16078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709fdd6e68714c09983375ccf1e9b268" candidate_uuid: "dfac52afbf744ee0969a49bd9dde53b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9579d1e3079a4ad29441c5c966335347" is_pre_election: true
I20260924 04:38:45.950199 16078 raft_consensus.cc:2454] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfac52afbf744ee0969a49bd9dde53b8 in term 0.
I20260924 04:38:45.950196 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709fdd6e68714c09983375ccf1e9b268" candidate_uuid: "dfac52afbf744ee0969a49bd9dde53b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cf48e1833b403bb5546aebf0181f1c" is_pre_election: true
I20260924 04:38:45.950477 15944 raft_consensus.cc:2454] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfac52afbf744ee0969a49bd9dde53b8 in term 0.
I20260924 04:38:45.950670 16147 leader_election.cc:302] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9579d1e3079a4ad29441c5c966335347, dfac52afbf744ee0969a49bd9dde53b8; no voters:
I20260924 04:38:45.950978 16283 raft_consensus.cc:2781] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:38:45.951076 16283 raft_consensus.cc:482] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:38:45.951136 16283 raft_consensus.cc:3037] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:45.952150 16283 raft_consensus.cc:504] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:38:45.952483 16283 leader_election.cc:288] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [CANDIDATE]: Term 1 election: Requested vote from peers 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097), b6cf48e1833b403bb5546aebf0181f1c (127.0.98.1:40195)
I20260924 04:38:45.952858 16078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709fdd6e68714c09983375ccf1e9b268" candidate_uuid: "dfac52afbf744ee0969a49bd9dde53b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9579d1e3079a4ad29441c5c966335347"
I20260924 04:38:45.952904 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709fdd6e68714c09983375ccf1e9b268" candidate_uuid: "dfac52afbf744ee0969a49bd9dde53b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cf48e1833b403bb5546aebf0181f1c"
I20260924 04:38:45.952997 15944 raft_consensus.cc:3037] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:45.952988 16078 raft_consensus.cc:3037] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:38:45.954365 15944 raft_consensus.cc:2454] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfac52afbf744ee0969a49bd9dde53b8 in term 1.
I20260924 04:38:45.954365 16078 raft_consensus.cc:2454] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfac52afbf744ee0969a49bd9dde53b8 in term 1.
I20260924 04:38:45.954671 16147 leader_election.cc:302] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9579d1e3079a4ad29441c5c966335347, dfac52afbf744ee0969a49bd9dde53b8; no voters:
I20260924 04:38:45.954859 16283 raft_consensus.cc:2781] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:38:45.955125 16283 raft_consensus.cc:686] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 1 LEADER]: Becoming Leader. State: Replica: dfac52afbf744ee0969a49bd9dde53b8, State: Running, Role: LEADER
I20260924 04:38:45.955482 16283 consensus_queue.cc:237] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:38:45.958736 15797 catalog_manager.cc:5495] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dfac52afbf744ee0969a49bd9dde53b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } health_report { overall_health: HEALTHY } } }
I20260924 04:38:45.972630 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:45.969168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.0.98.3:38973)
0924 04:38:45.969299 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:38:45.969303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:38:45.969317 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:38:45.969450 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:38:45.969452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:38:45.969507 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:38:45.969597 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:38:45.969851 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.970361 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.971211 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:38:45.971420 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:38:45.971502 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:38:45.971738 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:38:45.972151 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:38:45.972244 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:38:45.972372 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 04:38:45.972423 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:38:45.978931 16078 raft_consensus.cc:1261] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [term 1 FOLLOWER]: Refusing update from remote peer dfac52afbf744ee0969a49bd9dde53b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:45.978931 15944 raft_consensus.cc:1261] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Refusing update from remote peer dfac52afbf744ee0969a49bd9dde53b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:38:45.979714 16283 consensus_queue.cc:1035] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:45.979945 16285 consensus_queue.cc:1035] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:38:45.990420 16289 mvcc.cc:205] Tried to move back new op lower bound from 7332760477601558528 to 7332760477516271616. Current Snapshot: MvccSnapshot[applied={T|T < 7332760477601558528}]
I20260924 04:38:45.995656 15797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:38:45.995884 15797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:38:45.998319 15797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Sending DeleteTablet for 3 replicas of tablet 709fdd6e68714c09983375ccf1e9b268
I20260924 04:38:45.999034 16058 tablet_service.cc:1514] Processing DeleteTablet for tablet 709fdd6e68714c09983375ccf1e9b268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38010
I20260924 04:38:45.999155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 15861
I20260924 04:38:45.999181 15924 tablet_service.cc:1514] Processing DeleteTablet for tablet 709fdd6e68714c09983375ccf1e9b268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35348
I20260924 04:38:46.000123 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet 709fdd6e68714c09983375ccf1e9b268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33254
I20260924 04:38:46.000746 16297 tablet_replica.cc:330] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: stopping tablet replica
I20260924 04:38:46.001124 16297 raft_consensus.cc:2229] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:38:46.001282 16298 tablet_replica.cc:330] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: stopping tablet replica
I20260924 04:38:46.001577 16298 raft_consensus.cc:2229] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:38:46.001600 16297 raft_consensus.cc:2258] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:46.001922 16298 raft_consensus.cc:2258] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:38:46.003273 16297 ts_tablet_manager.cc:1905] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:38:46.003665 16298 ts_tablet_manager.cc:1905] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:38:46.006467 16298 ts_tablet_manager.cc:1918] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:38:46.006574 16298 log.cc:1196] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/709fdd6e68714c09983375ccf1e9b268
I20260924 04:38:46.006928 16298 ts_tablet_manager.cc:1939] T 709fdd6e68714c09983375ccf1e9b268 P dfac52afbf744ee0969a49bd9dde53b8: Deleting consensus metadata
I20260924 04:38:46.007103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 15995
W20260924 04:38:46.007103 15785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40195 (error 108)
I20260924 04:38:46.009322 15785 catalog_manager.cc:4841] TS dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973): tablet 709fdd6e68714c09983375ccf1e9b268 (table test-table [id=cf453024494845c382157b2dac74abed]) successfully deleted
I20260924 04:38:46.009969 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:46.009663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40195 (local address 127.0.0.1:35352)
0924 04:38:46.009808 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:46.009897 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 04:38:46.010234 15785 catalog_manager.cc:4584] TS b6cf48e1833b403bb5546aebf0181f1c (127.0.98.1:40195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 709fdd6e68714c09983375ccf1e9b268: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40195: connect: Connection refused (error 111)
I20260924 04:38:46.013264 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:38:46.012922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34097 (local address 127.0.0.1:38020)
0924 04:38:46.013108 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 04:38:46.013184 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 04:38:46.013415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 16129
I20260924 04:38:46.018527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 15767
I20260924 04:39:06.025265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36685
--webserver_interface=127.0.98.62
--webserver_port=39137
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:06.133432 16300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:06.133662 16300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:06.133700 16300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:06.137634 16300 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:39:06.137676 16300 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:39:06.137691 16300 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:39:06.137704 16300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:06.137758 16300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:06.137792 16300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:06.137822 16300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:06.137861 16300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:06.142938 16300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:06.143929 16300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:06.144919 16300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:39:06.157516 16300 server_base.cc:1034] running on GCE node
W20260924 04:39:06.157486 16309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:06.157532 16307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:06.157469 16306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:06.158440 16300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:06.159536 16300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:06.160756 16300 hybrid_clock.cc:648] HybridClock initialized: now 1790224746160700 us; error 49 us; skew 500 ppm
Sep 24 04:39:06 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:06.163537 16300 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:39:06.164402 16300 webserver.cc:460] Webserver started at http://127.0.98.62:39137/ using document root <none> and password file <none>
I20260924 04:39:06.164965 16300 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:06.165066 16300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:06.168833 16300 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:06.171103 16316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:06.172338 16300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:06.172477 16300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "4a66bd231e604fada64bec2070732cc3"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "34d867946af5e3d3f66005d78b853ebb"
server_key_iv: "62212f89f41dbf6f0f856cea44ac29ad"
server_key_version: "encryptionkey@0"
I20260924 04:39:06.173058 16300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:06.191531 16300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:06.194417 16300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:06.194612 16300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:06.202471 16300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36685
I20260924 04:39:06.202777 16368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36685 every 8 connection(s)
I20260924 04:39:06.203608 16300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:06.207907 16369 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:39:06.209523 16369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Bootstrap starting.
I20260924 04:39:06.211524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 16300
I20260924 04:39:06.211988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40195
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36685
--webserver_port=35441
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:39:06.226802 16369 log.cc:824] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:06.235112 16369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:39:06.235477 16369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Bootstrap complete.
I20260924 04:39:06.239473 16369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:39:06.239849 16369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a66bd231e604fada64bec2070732cc3, State: Initialized, Role: FOLLOWER
I20260924 04:39:06.240357 16369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:39:06.240474 16369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:06.240558 16369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:06.240662 16369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:39:06.242501 16369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:39:06.242815 16369 leader_election.cc:302] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a66bd231e604fada64bec2070732cc3; no voters:
I20260924 04:39:06.243116 16369 leader_election.cc:288] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:39:06.243192 16373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:39:06.243418 16373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 2 LEADER]: Becoming Leader. State: Replica: 4a66bd231e604fada64bec2070732cc3, State: Running, Role: LEADER
I20260924 04:39:06.243697 16373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:39:06.244396 16369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:06.244686 16373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4a66bd231e604fada64bec2070732cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } } }
I20260924 04:39:06.244800 16373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:06.245215 16377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:06.245301 16374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a66bd231e604fada64bec2070732cc3. Latest consensus state: current_term: 2 leader_uuid: "4a66bd231e604fada64bec2070732cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } } }
I20260924 04:39:06.245476 16374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:06.249861 16377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:06.250494 16377 catalog_manager.cc:1259] Loaded cluster ID: 303a3da5cf544fe7b821f470c9f64ab0
I20260924 04:39:06.250566 16377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:06.252228 16377 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:06.252797 16377 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Loaded TSK: 0
I20260924 04:39:06.272280 16377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Generated new TSK 1
I20260924 04:39:06.272540 16377 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:39:06.334729 16371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:06.334956 16371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:06.334996 16371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:06.338943 16371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:06.338994 16371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:06.339080 16371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:39:06.344197 16371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35441
--tserver_master_addrs=127.0.98.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:06.345122 16371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:06.346201 16371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:06.356671 16397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:06.356710 16396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:06.356854 16399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:06.357371 16371 server_base.cc:1034] running on GCE node
I20260924 04:39:06.357733 16371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:06.358318 16371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:06.359499 16371 hybrid_clock.cc:648] HybridClock initialized: now 1790224746359469 us; error 55 us; skew 500 ppm
Sep 24 04:39:06 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:06.362357 16371 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:39:06.363365 16371 webserver.cc:460] Webserver started at http://127.0.98.1:35441/ using document root <none> and password file <none>
I20260924 04:39:06.364009 16371 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:06.364104 16371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:06.368888 16371 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:39:06.371395 16406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:06.372812 16371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:39:06.372925 16371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "b6cf48e1833b403bb5546aebf0181f1c"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "ecc859cb04aa72cd9105b41bf79dfbee"
server_key_iv: "78f0ec6e5ef442477c295ba268f48c01"
server_key_version: "encryptionkey@0"
I20260924 04:39:06.373643 16371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:06.408128 16371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:06.411741 16371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:06.411934 16371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:06.412549 16371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:06.413784 16413 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:39:06.417590 16371 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:39:06.417647 16371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260924 04:39:06.417693 16371 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:39:06.419237 16371 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:39:06.419278 16371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:39:06.419610 16413 tablet_bootstrap.cc:492] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Bootstrap starting.
I20260924 04:39:06.429697 16371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40195
I20260924 04:39:06.429800 16520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40195 every 8 connection(s)
I20260924 04:39:06.430595 16371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:39:06.438726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 16371
I20260924 04:39:06.439128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34097
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36685
--webserver_port=32873
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:39:06.439875 16413 log.cc:824] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:39:06 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:06.447659 16413 tablet_bootstrap.cc:492] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:39:06.448019 16413 tablet_bootstrap.cc:492] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Bootstrap complete.
I20260924 04:39:06.449371 16413 ts_tablet_manager.cc:1397] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.015s
I20260924 04:39:06.449957 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.432696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:56723)
0924 04:39:06.433513 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:06.433588 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:06.434600 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:06.435651 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:06.435660 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:06.436213 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:06.436891 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.436910 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.438196 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.438199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.439469 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.439480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.440577 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.440584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.440803 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.441544 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:06.441564 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:06.443463 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:06.446368 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:06.446375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:06.446389 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:06.446763 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:06.447214 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:06.447217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:06.447219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:06.447376 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:06.448157 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:06.448163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:06.448453 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 04:39:06.448645 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 04:39:06.448977 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":142,"threads_started":1}
I20260924 04:39:06.450109 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.432723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56723 (local address 127.0.98.62:36685)
0924 04:39:06.433097 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:06.433116 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:06.433712 (+ 596us) server_negotiation.cc:407] Connection header received
0924 04:39:06.434992 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:06.435002 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:06.435265 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:06.435523 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:06.437074 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.438046 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.439599 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.440395 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.441226 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.443677 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:06.443685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:06.443702 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:06.443746 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:06.446202 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.446923 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.446931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.446939 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.447096 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.447490 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.447496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.447501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.447987 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:06.448158 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:06.448803 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 04:39:06.449120 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":114,"threads_started":1}
I20260924 04:39:06.452037 16413 raft_consensus.cc:348] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:39:06.452461 16413 raft_consensus.cc:729] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cf48e1833b403bb5546aebf0181f1c, State: Initialized, Role: FOLLOWER
I20260924 04:39:06.452826 16521 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36685
I20260924 04:39:06.452934 16413 consensus_queue.cc:260] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } }
I20260924 04:39:06.453040 16521 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:06.453567 16521 heartbeater.cc:511] Master 127.0.98.62:36685 requested a full tablet report, sending...
I20260924 04:39:06.453864 16413 ts_tablet_manager.cc:1428] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260924 04:39:06.455965 16333 ts_manager.cc:194] Registered new tserver with Master: b6cf48e1833b403bb5546aebf0181f1c (127.0.98.1:40195)
I20260924 04:39:06.460126 16333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56723
I20260924 04:39:06.462649 16521 heartbeater.cc:503] Master 127.0.98.62:36685 was elected leader, sending a full tablet report...
Sep 24 04:39:06 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:39:06.468463 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.459129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.0.98.1:40195)
0924 04:39:06.459789 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:06.459798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:06.459813 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:39:06.459859 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:06.459862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:06.460017 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:06.460154 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:06.460733 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.461479 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.462336 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.462662 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.462873 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.465272 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:06.465280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:06.465289 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:06.465309 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:06.466764 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.467270 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.467274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.467275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.467345 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.467605 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.467607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.467608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.467804 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:06.467920 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:06.468234 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 04:39:06.468351 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":101,"threads_started":1}
I20260924 04:39:06.468503 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.459093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40195 (local address 127.0.0.1:41596)
0924 04:39:06.459451 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:06.459491 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:06.459602 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:06.460156 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:06.460162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:06.460519 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:06.460616 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.460622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.461597 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.461602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.462217 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.462226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.462861 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.462877 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.463000 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.463600 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:06.463612 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:06.465131 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:06.466870 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:06.466878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:06.466884 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:06.467161 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:06.467427 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:06.467430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:06.467432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:06.467522 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:06.467909 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:06.467914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:06.468030 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 04:39:06.468169 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 04:39:06.468349 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":107,"threads_started":1}
I20260924 04:39:06.469543 16455 tablet_service.cc:1514] Processing DeleteTablet for tablet 709fdd6e68714c09983375ccf1e9b268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41596
I20260924 04:39:06.470366 16530 tablet_replica.cc:330] stopping tablet replica
I20260924 04:39:06.470657 16530 raft_consensus.cc:2229] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:06.470898 16530 raft_consensus.cc:2258] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:06.471902 16530 ts_tablet_manager.cc:1905] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:06.476155 16530 ts_tablet_manager.cc:1918] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:06.476244 16530 log.cc:1196] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal/wals/709fdd6e68714c09983375ccf1e9b268
I20260924 04:39:06.476517 16530 ts_tablet_manager.cc:1939] T 709fdd6e68714c09983375ccf1e9b268 P b6cf48e1833b403bb5546aebf0181f1c: Deleting consensus metadata
I20260924 04:39:06.478508 16319 catalog_manager.cc:4841] TS b6cf48e1833b403bb5546aebf0181f1c (127.0.98.1:40195): tablet 709fdd6e68714c09983375ccf1e9b268 (table test-table [id=cf453024494845c382157b2dac74abed]) successfully deleted
W20260924 04:39:06.556041 16526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:06.556273 16526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:06.556313 16526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:06.560220 16526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:06.560274 16526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:06.560330 16526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:39:06.565461 16526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32873
--tserver_master_addrs=127.0.98.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:06.566366 16526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:06.567265 16526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:06.576772 16535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:06.576777 16538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:06.576746 16536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:06.576924 16526 server_base.cc:1034] running on GCE node
I20260924 04:39:06.577493 16526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:06.578088 16526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:06.579274 16526 hybrid_clock.cc:648] HybridClock initialized: now 1790224746579233 us; error 55 us; skew 500 ppm
Sep 24 04:39:06 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:06.581952 16526 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:39:06.582795 16526 webserver.cc:460] Webserver started at http://127.0.98.2:32873/ using document root <none> and password file <none>
I20260924 04:39:06.583330 16526 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:06.583433 16526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:06.587080 16526 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:06.589172 16545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:06.590350 16526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:06.590489 16526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "9579d1e3079a4ad29441c5c966335347"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "b8a7102acb85cd698efe26d8d37f4900"
server_key_iv: "50e0520d468834fd1adaef6d14d4ed97"
server_key_version: "encryptionkey@0"
I20260924 04:39:06.590999 16526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:06.616103 16526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:06.618820 16526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:06.619017 16526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:06.619591 16526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:06.620608 16552 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 04:39:06.624169 16552 ts_tablet_manager.cc:1852] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 04:39:06.624553 16552 ts_tablet_manager.cc:1905] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:06.628450 16552 ts_tablet_manager.cc:1918] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:06.628540 16552 log.cc:1196] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/709fdd6e68714c09983375ccf1e9b268
I20260924 04:39:06.628877 16552 ts_tablet_manager.cc:1939] T 709fdd6e68714c09983375ccf1e9b268 P 9579d1e3079a4ad29441c5c966335347: Deleting consensus metadata
I20260924 04:39:06.629206 16526 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 04:39:06.629292 16526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260924 04:39:06.629371 16526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:06.629424 16526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:06.638082 16526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34097
I20260924 04:39:06.638121 16659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34097 every 8 connection(s)
I20260924 04:39:06.638881 16526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:39:06.645903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 16526
I20260924 04:39:06.646386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38973
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36685
--webserver_port=46315
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:39:06 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:06.652477 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.640581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54863 (local address 127.0.98.62:36685)
0924 04:39:06.640728 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:06.640731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:06.641550 (+ 819us) server_negotiation.cc:407] Connection header received
0924 04:39:06.642459 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:06.642462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:06.642526 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:06.642631 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:06.643775 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.644368 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.645623 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.645932 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.646046 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.648434 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:06.648438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:06.648440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:06.648473 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:06.650219 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.650790 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.650795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.650800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.650906 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.651238 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.651244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.651248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.651513 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:06.651628 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:06.652080 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 04:39:06.652228 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:39:06.653000 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.640935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:54863)
0924 04:39:06.641398 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:06.641437 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:06.642251 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:06.642796 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:06.642802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:06.643123 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:06.643634 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.643647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.644499 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.644504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.645498 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.645512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.646078 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.646082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.646288 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.646888 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:06.646903 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:06.648269 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:06.650338 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:06.650342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:06.650353 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:06.650681 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:06.651010 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:06.651012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:06.651013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:06.651139 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:06.651619 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:06.651624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:06.651827 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 04:39:06.651995 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:39:06.652218 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260924 04:39:06.654161 16660 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36685
I20260924 04:39:06.654388 16660 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:06.654829 16660 heartbeater.cc:511] Master 127.0.98.62:36685 requested a full tablet report, sending...
I20260924 04:39:06.655829 16333 ts_manager.cc:194] Registered new tserver with Master: 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097)
I20260924 04:39:06.656370 16333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54863
W20260924 04:39:06.754933 16663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:06.755157 16663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:06.755227 16663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:06.759171 16663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:06.759222 16663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:06.759322 16663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:39:06.764343 16663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46315
--tserver_master_addrs=127.0.98.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:06.765342 16663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:06.766291 16663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:06.775591 16669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:06.775586 16668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:06.775662 16663 server_base.cc:1034] running on GCE node
W20260924 04:39:06.775584 16671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:06.776302 16663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:06.776969 16663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:06.778153 16663 hybrid_clock.cc:648] HybridClock initialized: now 1790224746778139 us; error 34 us; skew 500 ppm
Sep 24 04:39:06 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:06.780869 16663 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:39:06.781737 16663 webserver.cc:460] Webserver started at http://127.0.98.3:46315/ using document root <none> and password file <none>
I20260924 04:39:06.782256 16663 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:06.782369 16663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:06.785940 16663 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:06.787883 16678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:06.788985 16663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:06.789098 16663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "dfac52afbf744ee0969a49bd9dde53b8"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "5b7cf01153c577ba9356e8a47de7738e"
server_key_iv: "8caec76807069b445f7a5dafaadee133"
server_key_version: "encryptionkey@0"
I20260924 04:39:06.789651 16663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:06.803881 16663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:06.806679 16663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:06.806872 16663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:06.807416 16663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:06.808168 16663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:06.808219 16663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:06.808255 16663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:06.808306 16663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:06.817289 16663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38973
I20260924 04:39:06.817315 16791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38973 every 8 connection(s)
I20260924 04:39:06.818225 16663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:39:06.822916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 16663
Sep 24 04:39:06 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:06.832340 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.823597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.0.98.62:36685)
0924 04:39:06.823907 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:06.823912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:06.824081 (+ 169us) server_negotiation.cc:407] Connection header received
0924 04:39:06.824123 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:06.824125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:06.824164 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:06.824237 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:06.825012 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.825773 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.826835 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.827194 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.827308 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.828623 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:06.828630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:06.828634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:06.828664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:06.830511 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.831009 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.831013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.831015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.831100 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.831399 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.831401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.831402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.831624 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:06.831822 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:06.832056 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 04:39:06.832170 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":87,"threads_started":1}
I20260924 04:39:06.832830 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.819896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50301 (local address 127.0.98.62:36685)
0924 04:39:06.820030 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:06.820034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:06.820802 (+ 768us) server_negotiation.cc:407] Connection header received
0924 04:39:06.822052 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:06.822055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:06.822111 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:06.822218 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:06.823496 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.824199 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.825777 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.826094 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.826199 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.828713 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:06.828716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:06.828717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:06.828751 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:06.830712 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.831280 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.831283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.831285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.831369 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.831730 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.831734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.831735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.831928 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:06.832020 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:06.832579 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 04:39:06.832679 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:39:06.833770 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.820238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:50301)
0924 04:39:06.820650 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:06.820689 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:06.821848 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:06.822386 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:06.822392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:06.822757 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:06.823297 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.823317 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.824396 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.824401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.825648 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.825659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.826241 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.826244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.826378 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.826984 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:06.827002 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:06.828522 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:06.830827 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:06.830832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:06.830842 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:06.831186 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:06.831460 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:06.831467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:06.831470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:06.831624 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:06.832032 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:06.832039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:06.832330 (+ 291us) client_negotiation.cc:769] Sending connection context
0924 04:39:06.832529 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 04:39:06.832827 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
I20260924 04:39:06.834822 16792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36685
I20260924 04:39:06.835050 16792 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:06.835502 16792 heartbeater.cc:511] Master 127.0.98.62:36685 requested a full tablet report, sending...
I20260924 04:39:06.836391 16333 ts_manager.cc:194] Registered new tserver with Master: dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973)
I20260924 04:39:06.836872 16333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50301
I20260924 04:39:06.846048 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:39:06.849581 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.846640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.0.98.62:36685)
0924 04:39:06.846749 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:06.846752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:06.846956 (+ 204us) server_negotiation.cc:407] Connection header received
0924 04:39:06.847072 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:06.847074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:06.847155 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:06.847232 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:39:06.847422 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.847877 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.848657 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.848865 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.848935 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.849215 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:39:06.849321 (+ 106us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 04:39:06.849501 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35406: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 04:39:06.849653 16523 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35406: token expired
W20260924 04:39:06.849990 16263 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:36685)
I20260924 04:39:06.856350 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.850329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.0.98.62:36685)
0924 04:39:06.850445 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:06.850448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:06.850459 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:39:06.850507 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:06.850509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:06.850539 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:06.850598 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:06.851149 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.851582 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.852323 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.852508 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.852575 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.853724 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:06.853727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:06.853729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:06.853748 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:06.855150 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.855501 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.855503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.855504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.855570 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:06.855777 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:06.855779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:06.855780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:06.855935 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:06.856028 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:06.856191 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 04:39:06.856263 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:39:06.858639 16333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:39:06.861696 16333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:39:06.869213 16455 tablet_service.cc:1467] Processing CreateTablet for tablet a1953c4fb8ef46328653096b8311ae33 (DEFAULT_TABLE table=test-table [id=b8869eb4c5d94cdba90269deb2cba65c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:06.869874 16455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1953c4fb8ef46328653096b8311ae33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:06.872888 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.868066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34097 (local address 127.0.0.1:53460)
0924 04:39:06.868187 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:06.868201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:06.868283 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:06.869066 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:06.869069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:06.869086 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:06.869205 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.869212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.870357 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.870363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.871325 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.871335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.872476 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.872479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.872566 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.872603 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 04:39:06.872686 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:39:06.872778 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:39:06.872989 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.868240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.0.98.2:34097)
0924 04:39:06.868582 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:06.868594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:06.868618 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:39:06.868697 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:06.868705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:06.868925 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:06.869087 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:06.869303 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.870253 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.871459 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.872361 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.872478 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.872614 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:06.872752 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 04:39:06.872838 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":116,"threads_started":1}
I20260924 04:39:06.873101 16413 tablet_bootstrap.cc:492] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Bootstrap starting.
I20260924 04:39:06.873826 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.868279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38973 (local address 127.0.0.1:52296)
0924 04:39:06.868645 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:06.868658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:06.868735 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:06.869480 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:06.869483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:06.869518 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:06.869613 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.869619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.871109 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.871112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.872243 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:06.872254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.873401 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.873404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:06.873490 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.873512 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:39:06.873591 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:39:06.873654 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":107,"threads_started":1}
I20260924 04:39:06.874056 16413 tablet_bootstrap.cc:654] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:06.874231 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:06.868583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.0.98.3:38973)
0924 04:39:06.869000 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:06.869011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:06.869032 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:39:06.869098 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:06.869103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:06.869322 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:06.869775 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:06.869951 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.870994 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.872377 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:06.873274 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:06.873372 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:06.873497 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:06.873986 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 04:39:06.874090 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":167,"threads_started":1}
I20260924 04:39:06.874897 16594 tablet_service.cc:1467] Processing CreateTablet for tablet a1953c4fb8ef46328653096b8311ae33 (DEFAULT_TABLE table=test-table [id=b8869eb4c5d94cdba90269deb2cba65c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:06.875494 16413 tablet_bootstrap.cc:492] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: No bootstrap required, opened a new log
I20260924 04:39:06.875574 16413 ts_tablet_manager.cc:1397] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 04:39:06.875892 16594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1953c4fb8ef46328653096b8311ae33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:06.875967 16413 raft_consensus.cc:348] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:06.876085 16413 raft_consensus.cc:374] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:06.876127 16413 raft_consensus.cc:729] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cf48e1833b403bb5546aebf0181f1c, State: Initialized, Role: FOLLOWER
I20260924 04:39:06.876247 16413 consensus_queue.cc:260] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:06.876927 16413 ts_tablet_manager.cc:1428] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:39:06.877415 16726 tablet_service.cc:1467] Processing CreateTablet for tablet a1953c4fb8ef46328653096b8311ae33 (DEFAULT_TABLE table=test-table [id=b8869eb4c5d94cdba90269deb2cba65c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:06.878437 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1953c4fb8ef46328653096b8311ae33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:06.880851 16552 tablet_bootstrap.cc:492] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: Bootstrap starting.
I20260924 04:39:06.882997 16552 tablet_bootstrap.cc:654] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:06.883826 16552 log.cc:824] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:06.885648 16552 tablet_bootstrap.cc:492] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: No bootstrap required, opened a new log
I20260924 04:39:06.885869 16552 ts_tablet_manager.cc:1397] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:39:06.886154 16806 tablet_bootstrap.cc:492] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: Bootstrap starting.
I20260924 04:39:06.888799 16806 tablet_bootstrap.cc:654] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:06.889833 16806 log.cc:824] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:06.889803 16552 raft_consensus.cc:348] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:06.890051 16552 raft_consensus.cc:374] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:06.890120 16552 raft_consensus.cc:729] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9579d1e3079a4ad29441c5c966335347, State: Initialized, Role: FOLLOWER
I20260924 04:39:06.890614 16552 consensus_queue.cc:260] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:06.891279 16660 heartbeater.cc:503] Master 127.0.98.62:36685 was elected leader, sending a full tablet report...
I20260924 04:39:06.891604 16552 ts_tablet_manager.cc:1428] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:39:06.891741 16806 tablet_bootstrap.cc:492] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: No bootstrap required, opened a new log
I20260924 04:39:06.891923 16806 ts_tablet_manager.cc:1397] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:39:06.895042 16806 raft_consensus.cc:348] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:06.895285 16806 raft_consensus.cc:374] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:06.895321 16806 raft_consensus.cc:729] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfac52afbf744ee0969a49bd9dde53b8, State: Initialized, Role: FOLLOWER
I20260924 04:39:06.895673 16806 consensus_queue.cc:260] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:06.896195 16792 heartbeater.cc:503] Master 127.0.98.62:36685 was elected leader, sending a full tablet report...
I20260924 04:39:06.896348 16806 ts_tablet_manager.cc:1428] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260924 04:39:06.931695 16522 tablet.cc:2369] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:39:07.069200 16793 tablet.cc:2369] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:39:07.139976 16661 tablet.cc:2369] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:39:07.180833 16527 raft_consensus.cc:482] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:07.181095 16527 raft_consensus.cc:504] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:07.182301 16527 leader_election.cc:288] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097), dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973)
I20260924 04:39:07.187503 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:07.182963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39957 (local address 127.0.98.3:38973)
0924 04:39:07.183080 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:07.183085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:07.183235 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:39:07.183391 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:07.183394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:07.183446 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:07.183527 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:07.183821 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.184543 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.186135 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.186762 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.186915 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:07.186962 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:07.187291 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 04:39:07.187373 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:39:07.187503 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:07.182717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38517 (local address 127.0.98.2:34097)
0924 04:39:07.182852 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:07.182857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:07.183011 (+ 154us) server_negotiation.cc:407] Connection header received
0924 04:39:07.183210 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:07.183213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:07.183267 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:07.183366 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:07.183620 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.184436 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.186135 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.186761 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.186892 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:07.186960 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:07.187291 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 04:39:07.187379 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:39:07.187573 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:07.182490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34097 (local address 127.0.98.1:38517)
0924 04:39:07.182923 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:07.182934 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:07.183082 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:07.183379 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:07.183382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:07.183399 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:07.183511 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:07.183516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.184547 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.184549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:07.185997 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:07.186007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.186893 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.186897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:07.187072 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:07.187181 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 04:39:07.187260 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:39:07.187391 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":156,"thread_start_us":83,"threads_started":1}
I20260924 04:39:07.187573 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:07.182867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38973 (local address 127.0.98.1:39957)
0924 04:39:07.183148 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:07.183160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:07.183275 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:07.183628 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:07.183630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:07.183644 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:07.183710 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:07.183714 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.184641 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.184643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:07.185996 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:07.186007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.186906 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.186909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:07.187067 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:07.187179 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 04:39:07.187260 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:39:07.187391 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":191,"thread_start_us":57,"threads_started":1}
I20260924 04:39:07.188205 16614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1953c4fb8ef46328653096b8311ae33" candidate_uuid: "b6cf48e1833b403bb5546aebf0181f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9579d1e3079a4ad29441c5c966335347" is_pre_election: true
I20260924 04:39:07.188206 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1953c4fb8ef46328653096b8311ae33" candidate_uuid: "b6cf48e1833b403bb5546aebf0181f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfac52afbf744ee0969a49bd9dde53b8" is_pre_election: true
I20260924 04:39:07.188512 16614 raft_consensus.cc:2454] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cf48e1833b403bb5546aebf0181f1c in term 0.
I20260924 04:39:07.188512 16746 raft_consensus.cc:2454] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cf48e1833b403bb5546aebf0181f1c in term 0.
I20260924 04:39:07.189037 16409 leader_election.cc:302] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9579d1e3079a4ad29441c5c966335347, b6cf48e1833b403bb5546aebf0181f1c; no voters:
I20260924 04:39:07.189375 16527 raft_consensus.cc:2781] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:39:07.189472 16527 raft_consensus.cc:482] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:39:07.189517 16527 raft_consensus.cc:3037] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:07.190580 16527 raft_consensus.cc:504] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:07.190855 16527 leader_election.cc:288] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 1 election: Requested vote from peers 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097), dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973)
I20260924 04:39:07.191262 16614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1953c4fb8ef46328653096b8311ae33" candidate_uuid: "b6cf48e1833b403bb5546aebf0181f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9579d1e3079a4ad29441c5c966335347"
I20260924 04:39:07.191381 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1953c4fb8ef46328653096b8311ae33" candidate_uuid: "b6cf48e1833b403bb5546aebf0181f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfac52afbf744ee0969a49bd9dde53b8"
I20260924 04:39:07.191433 16614 raft_consensus.cc:3037] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:07.191555 16746 raft_consensus.cc:3037] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:07.192477 16614 raft_consensus.cc:2454] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cf48e1833b403bb5546aebf0181f1c in term 1.
I20260924 04:39:07.192629 16746 raft_consensus.cc:2454] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cf48e1833b403bb5546aebf0181f1c in term 1.
I20260924 04:39:07.192801 16409 leader_election.cc:302] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9579d1e3079a4ad29441c5c966335347, b6cf48e1833b403bb5546aebf0181f1c; no voters:
I20260924 04:39:07.192987 16527 raft_consensus.cc:2781] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:07.193167 16527 raft_consensus.cc:686] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 LEADER]: Becoming Leader. State: Replica: b6cf48e1833b403bb5546aebf0181f1c, State: Running, Role: LEADER
I20260924 04:39:07.193429 16527 consensus_queue.cc:237] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:07.195878 16332 catalog_manager.cc:5495] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c reported cstate change: term changed from 0 to 1, leader changed from <none> to b6cf48e1833b403bb5546aebf0181f1c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b6cf48e1833b403bb5546aebf0181f1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } health_report { overall_health: HEALTHY } } }
I20260924 04:39:07.239651 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:07.235901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.0.98.1:40195)
0924 04:39:07.236236 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:07.236243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:07.236254 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:39:07.236322 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:07.236324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:07.236376 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:07.236505 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:39:07.236774 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.237350 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.238183 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:07.238421 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:07.238536 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:07.238852 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:39:07.239251 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:39:07.239381 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:07.239439 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 04:39:07.239491 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260924 04:39:07.243834 16614 raft_consensus.cc:1261] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [term 1 FOLLOWER]: Refusing update from remote peer b6cf48e1833b403bb5546aebf0181f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:07.243861 16746 raft_consensus.cc:1261] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [term 1 FOLLOWER]: Refusing update from remote peer b6cf48e1833b403bb5546aebf0181f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:07.244508 16527 consensus_queue.cc:1035] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:07.244750 16812 consensus_queue.cc:1035] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:07.253152 16817 mvcc.cc:205] Tried to move back new op lower bound from 7332760564703158272 to 7332760564506574848. Current Snapshot: MvccSnapshot[applied={T|T < 7332760564703158272}]
I20260924 04:39:07.255244 16818 mvcc.cc:205] Tried to move back new op lower bound from 7332760564703158272 to 7332760564506574848. Current Snapshot: MvccSnapshot[applied={T|T < 7332760564703158272}]
I20260924 04:39:07.259024 16333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:07.259150 16333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:07.261310 16333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Sending DeleteTablet for 3 replicas of tablet a1953c4fb8ef46328653096b8311ae33
I20260924 04:39:07.261899 16594 tablet_service.cc:1514] Processing DeleteTablet for tablet a1953c4fb8ef46328653096b8311ae33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53460
I20260924 04:39:07.262034 16455 tablet_service.cc:1514] Processing DeleteTablet for tablet a1953c4fb8ef46328653096b8311ae33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41596
I20260924 04:39:07.262121 16726 tablet_service.cc:1514] Processing DeleteTablet for tablet a1953c4fb8ef46328653096b8311ae33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52296
I20260924 04:39:07.262358 16826 tablet_replica.cc:330] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: stopping tablet replica
I20260924 04:39:07.262488 16826 raft_consensus.cc:2229] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:39:07.262698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 16371
I20260924 04:39:07.262709 16828 tablet_replica.cc:330] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: stopping tablet replica
I20260924 04:39:07.262977 16828 raft_consensus.cc:2229] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:07.263248 16828 raft_consensus.cc:2258] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:07.262516 16827 tablet_replica.cc:330] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: stopping tablet replica
I20260924 04:39:07.264283 16827 raft_consensus.cc:2229] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:07.264564 16827 raft_consensus.cc:2258] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:07.264889 16828 ts_tablet_manager.cc:1905] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:07.266207 16827 ts_tablet_manager.cc:1905] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:07.267577 16828 ts_tablet_manager.cc:1918] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:07.267683 16828 log.cc:1196] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/a1953c4fb8ef46328653096b8311ae33
I20260924 04:39:07.268040 16828 ts_tablet_manager.cc:1939] T a1953c4fb8ef46328653096b8311ae33 P dfac52afbf744ee0969a49bd9dde53b8: Deleting consensus metadata
I20260924 04:39:07.268854 16827 ts_tablet_manager.cc:1918] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:07.268932 16827 log.cc:1196] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/a1953c4fb8ef46328653096b8311ae33
I20260924 04:39:07.269225 16827 ts_tablet_manager.cc:1939] T a1953c4fb8ef46328653096b8311ae33 P 9579d1e3079a4ad29441c5c966335347: Deleting consensus metadata
I20260924 04:39:07.269263 16319 catalog_manager.cc:4841] TS dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973): tablet a1953c4fb8ef46328653096b8311ae33 (table test-table [id=b8869eb4c5d94cdba90269deb2cba65c]) successfully deleted
W20260924 04:39:07.269845 16319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40195 (error 108)
I20260924 04:39:07.270654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 16526
I20260924 04:39:07.270676 16319 catalog_manager.cc:4841] TS 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097): tablet a1953c4fb8ef46328653096b8311ae33 (table test-table [id=b8869eb4c5d94cdba90269deb2cba65c]) successfully deleted
I20260924 04:39:07.270844 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:07.270545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40195 (local address 127.0.0.1:41614)
0924 04:39:07.270669 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:07.270748 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 04:39:07.272704 16319 catalog_manager.cc:4584] TS b6cf48e1833b403bb5546aebf0181f1c (127.0.98.1:40195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1953c4fb8ef46328653096b8311ae33: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40195: connect: Connection refused (error 111)
I20260924 04:39:07.276084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 16663
I20260924 04:39:07.281561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 16300
I20260924 04:39:27.288002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36685
--webserver_interface=127.0.98.62
--webserver_port=39137
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:27.394001 16830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:27.394237 16830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:27.394276 16830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:27.398269 16830 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:39:27.398315 16830 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:39:27.398329 16830 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:39:27.398343 16830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:27.398360 16830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:27.398418 16830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:27.398447 16830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:27.398486 16830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:27.403503 16830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:27.404484 16830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:27.405489 16830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:27.413662 16836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:27.413643 16838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:27.413663 16835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:27.413851 16830 server_base.cc:1034] running on GCE node
I20260924 04:39:27.414573 16830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:27.415571 16830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:27.416750 16830 hybrid_clock.cc:648] HybridClock initialized: now 1790224767416736 us; error 30 us; skew 500 ppm
Sep 24 04:39:27 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:27.419855 16830 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:39:27.421023 16830 webserver.cc:460] Webserver started at http://127.0.98.62:39137/ using document root <none> and password file <none>
I20260924 04:39:27.421764 16830 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:27.421850 16830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:27.427244 16830 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 04:39:27.429363 16845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:27.430567 16830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:39:27.430699 16830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "4a66bd231e604fada64bec2070732cc3"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "34d867946af5e3d3f66005d78b853ebb"
server_key_iv: "62212f89f41dbf6f0f856cea44ac29ad"
server_key_version: "encryptionkey@0"
I20260924 04:39:27.431227 16830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:27.467159 16830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:27.470328 16830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:27.470532 16830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:27.477515 16830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36685
I20260924 04:39:27.477815 16897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36685 every 8 connection(s)
I20260924 04:39:27.478634 16830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:27.482863 16898 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:39:27.484349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 16830
I20260924 04:39:27.484637 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Bootstrap starting.
I20260924 04:39:27.484824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40195
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36685
--webserver_port=35441
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:39:27.502025 16898 log.cc:824] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:27.513119 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:39:27.513514 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Bootstrap complete.
I20260924 04:39:27.517356 16898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:39:27.517745 16898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a66bd231e604fada64bec2070732cc3, State: Initialized, Role: FOLLOWER
I20260924 04:39:27.518249 16898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:39:27.518389 16898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:27.518474 16898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:27.518586 16898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:39:27.520335 16898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:39:27.520648 16898 leader_election.cc:302] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a66bd231e604fada64bec2070732cc3; no voters:
I20260924 04:39:27.520978 16898 leader_election.cc:288] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:39:27.521055 16902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:39:27.521270 16902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [term 3 LEADER]: Becoming Leader. State: Replica: 4a66bd231e604fada64bec2070732cc3, State: Running, Role: LEADER
I20260924 04:39:27.521553 16902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } }
I20260924 04:39:27.522145 16898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:27.522435 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a66bd231e604fada64bec2070732cc3. Latest consensus state: current_term: 3 leader_uuid: "4a66bd231e604fada64bec2070732cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } } }
I20260924 04:39:27.522560 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:27.522811 16903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4a66bd231e604fada64bec2070732cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a66bd231e604fada64bec2070732cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36685 } } }
I20260924 04:39:27.522908 16903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:27.522910 16906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:27.527844 16906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:27.528478 16906 catalog_manager.cc:1259] Loaded cluster ID: 303a3da5cf544fe7b821f470c9f64ab0
I20260924 04:39:27.528894 16906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:27.531430 16906 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:27.532135 16906 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Loaded TSK: 0
I20260924 04:39:27.532219 16906 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a66bd231e604fada64bec2070732cc3: Loaded TSK: 1
F20260924 04:39:27.532941 16906 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe1164bd9ad google::LogMessage::Fail() at ??:0
@ 0x7fe1164c088a google::LogMessage::SendToLog() at ??:0
@ 0x7fe1164bd4de google::LogMessage::Flush() at ??:0
@ 0x7fe1164bea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe11918118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe119179f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe120e227bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe120dfc99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe120dfd178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fe120e3edf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe120e5d82b std::function<>::operator()() at ??:0
@ 0x7fe120dfcb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe120dfdf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe120dfa579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe120e3e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe11870f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe11870fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe1187113f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe118700512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe119d446db start_thread at ??:0
@ 0x7fe114fbb61f clone at ??:0
W20260924 04:39:27.611923 16900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:27.612133 16900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:27.612171 16900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:27.616410 16900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:27.616451 16900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:27.616506 16900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:39:27.622131 16900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35441
--tserver_master_addrs=127.0.98.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:27.623047 16900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:27.623960 16900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:27.633658 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:27.633628 16944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:27.633790 16943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:27.634102 16900 server_base.cc:1034] running on GCE node
I20260924 04:39:27.634521 16900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:27.635166 16900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:27.636335 16900 hybrid_clock.cc:648] HybridClock initialized: now 1790224767636300 us; error 47 us; skew 500 ppm
Sep 24 04:39:27 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:27.639478 16900 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:39:27.640506 16900 webserver.cc:460] Webserver started at http://127.0.98.1:35441/ using document root <none> and password file <none>
I20260924 04:39:27.641247 16900 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:27.641335 16900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:27.646234 16900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:39:27.648895 16955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:27.650430 16900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:39:27.650557 16900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "b6cf48e1833b403bb5546aebf0181f1c"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "ecc859cb04aa72cd9105b41bf79dfbee"
server_key_iv: "78f0ec6e5ef442477c295ba268f48c01"
server_key_version: "encryptionkey@0"
I20260924 04:39:27.651247 16900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:27.687140 16900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:27.690362 16900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:27.690553 16900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:27.691155 16900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:27.692231 16962 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:39:27.697512 16900 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:39:27.697587 16900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:39:27.697686 16900 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:39:27.699252 16900 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:39:27.699319 16900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:39:27.699608 16962 tablet_bootstrap.cc:492] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Bootstrap starting.
I20260924 04:39:27.709823 16900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40195
I20260924 04:39:27.709867 17069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40195 every 8 connection(s)
I20260924 04:39:27.710736 16900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:39:27.711515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 16900
I20260924 04:39:27.711927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34097
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36685
--webserver_port=32873
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:39:27.721436 16962 log.cc:824] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:27.730237 16962 tablet_bootstrap.cc:492] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:39:27.730681 16962 tablet_bootstrap.cc:492] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Bootstrap complete.
I20260924 04:39:27.732120 16962 ts_tablet_manager.cc:1397] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.008s
I20260924 04:39:27.735285 16962 raft_consensus.cc:348] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:27.735818 16962 raft_consensus.cc:729] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cf48e1833b403bb5546aebf0181f1c, State: Initialized, Role: FOLLOWER
I20260924 04:39:27.736336 16962 consensus_queue.cc:260] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:27.737391 16962 ts_tablet_manager.cc:1428] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
W20260924 04:39:27.786113 17073 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:39:27.713772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:43161)
0924 04:39:27.715050 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:27.715106 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:27.716312 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:27.785184 (+ 68872us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":162,"threads_started":1}
W20260924 04:39:27.786643 17070 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36685 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36685: Client connection negotiation failed: client connection to 127.0.98.62:36685: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:39:27.787391 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:27.787107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:39717)
0924 04:39:27.787247 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:27.787323 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:39:27.788077 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:27.787844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:56869)
0924 04:39:27.787967 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:27.788024 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:39:27.788275 17070 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:39:27.823083 17072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:27.823309 17072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:27.823380 17072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:27.827491 17072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:27.827541 17072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:27.827638 17072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:39:27.832664 17072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32873
--tserver_master_addrs=127.0.98.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:27.833604 17072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:27.834527 17072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:27.843786 17081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:27.843820 17080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:27.843786 17083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:27.843966 17072 server_base.cc:1034] running on GCE node
I20260924 04:39:27.844429 17072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:27.845067 17072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:27.846266 17072 hybrid_clock.cc:648] HybridClock initialized: now 1790224767846244 us; error 31 us; skew 500 ppm
Sep 24 04:39:27 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:27.849061 17072 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:39:27.849886 17072 webserver.cc:460] Webserver started at http://127.0.98.2:32873/ using document root <none> and password file <none>
I20260924 04:39:27.850420 17072 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:27.850510 17072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:27.854106 17072 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:27.856103 17090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:27.857394 17072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:27.857527 17072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "9579d1e3079a4ad29441c5c966335347"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "b8a7102acb85cd698efe26d8d37f4900"
server_key_iv: "50e0520d468834fd1adaef6d14d4ed97"
server_key_version: "encryptionkey@0"
I20260924 04:39:27.858062 17072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:27.874801 17072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:27.878047 17072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:27.878242 17072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:27.878818 17072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:27.879604 17072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:27.879650 17072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:27.879712 17072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:27.879745 17072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:27.888729 17072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34097
I20260924 04:39:27.888795 17203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34097 every 8 connection(s)
I20260924 04:39:27.889791 17072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:39:27.893432 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:27.891755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:38363)
0924 04:39:27.892215 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:27.892693 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":109,"threads_started":1}
W20260924 04:39:27.893887 17204 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36685 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36685: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
I20260924 04:39:27.894398 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:27.894207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:56451)
0924 04:39:27.894302 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:27.894347 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 04:39:27.895002 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:27.894795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:54309)
0924 04:39:27.894909 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:27.894954 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 04:39:27.895186 17204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:39:27.898862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 17072
I20260924 04:39:27.899266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38973
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36685
--webserver_port=46315
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:28.005235 17207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:28.005481 17207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:28.005522 17207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:28.009433 17207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:28.009482 17207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:28.009538 17207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:39:28.014617 17207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46315
--tserver_master_addrs=127.0.98.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:28.015524 17207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:28.016448 17207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:28.025498 17215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:28.025493 17213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:28.025594 17212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:28.025830 17207 server_base.cc:1034] running on GCE node
I20260924 04:39:28.026309 17207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:28.026964 17207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:28.028155 17207 hybrid_clock.cc:648] HybridClock initialized: now 1790224768028143 us; error 29 us; skew 500 ppm
Sep 24 04:39:28 dist-test-slave-jlnt krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:28.030972 17207 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:39:28.031755 17207 webserver.cc:460] Webserver started at http://127.0.98.3:46315/ using document root <none> and password file <none>
I20260924 04:39:28.032282 17207 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:28.032369 17207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:28.036047 17207 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:28.037983 17222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:28.039099 17207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:28.039225 17207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "dfac52afbf744ee0969a49bd9dde53b8"
format_stamp: "Formatted at 2026-09-24 04:38:45 on dist-test-slave-jlnt"
server_key: "5b7cf01153c577ba9356e8a47de7738e"
server_key_iv: "8caec76807069b445f7a5dafaadee133"
server_key_version: "encryptionkey@0"
I20260924 04:39:28.039748 17207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:28.051242 17207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:28.054178 17207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:28.054311 17207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:28.054754 17207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:28.055444 17207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:28.055488 17207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:28.055523 17207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:28.055544 17207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:28.063885 17207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38973
I20260924 04:39:28.063977 17335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38973 every 8 connection(s)
I20260924 04:39:28.064802 17207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:39:28.068302 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:28.066765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:54525)
0924 04:39:28.067186 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:28.067663 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
W20260924 04:39:28.068815 17336 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36685 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36685: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
I20260924 04:39:28.069495 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:28.069286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:46347)
0924 04:39:28.069382 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:28.069426 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 04:39:28.070073 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:28.069895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:44915)
0924 04:39:28.069989 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:28.070029 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 04:39:28.070256 17336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:39:28.075088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 17207
I20260924 04:39:28.789302 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:28.788808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:56525)
0924 04:39:28.789085 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:28.789184 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":71,"threads_started":1}
I20260924 04:39:28.896126 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:28.895774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:53491)
0924 04:39:28.895993 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:28.896042 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260924 04:39:29.071172 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:29.070771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:43161)
0924 04:39:29.071025 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:29.071079 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":69,"threads_started":1}
I20260924 04:39:29.226236 17343 raft_consensus.cc:482] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:29.226503 17343 raft_consensus.cc:504] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:29.227571 17343 leader_election.cc:288] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097), dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973)
Sep 24 04:39:29 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 04:39:29 dist-test-slave-jlnt krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 04:39:29.241703 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:29.227870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34097 (local address 127.0.98.1:50713)
0924 04:39:29.228091 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:29.228109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:29.228227 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:29.230180 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:29.230186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:29.230710 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:29.231156 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:29.231169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:29.232685 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:29.232688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:29.233692 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:29.233701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:29.234388 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:29.234391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:29.234620 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:29.235323 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:29.235335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:29.236905 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:29.239125 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:29.239131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:29.239144 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:29.239485 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:29.239822 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:29.239825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:29.239827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:29.240005 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:29.240722 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:29.240726 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:29.240950 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 04:39:29.241193 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 04:39:29.241517 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":171,"spinlock_wait_cycles":2816}
I20260924 04:39:29.241803 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:29.228482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35497 (local address 127.0.98.3:38973)
0924 04:39:29.228850 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:29.228869 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:29.228898 (+ 29us) server_negotiation.cc:407] Connection header received
0924 04:39:29.229570 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:29.229576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:29.229885 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:29.230160 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:29.231327 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:29.232566 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:29.233817 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:29.234266 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:29.234437 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:29.237508 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:29.237517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:29.237534 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:29.237568 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:29.239591 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:29.240043 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:29.240067 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:29.240074 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:29.240215 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:29.240471 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:29.240474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:29.240476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:29.240915 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:29.241017 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:29.241327 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 04:39:29.241611 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":74,"threads_started":1}
I20260924 04:39:29.241839 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:29.228243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38973 (local address 127.0.98.1:35497)
0924 04:39:29.228571 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:29.228587 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:29.228690 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:29.230178 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:29.230184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:29.230733 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:29.231156 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:29.231170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:29.232691 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:29.232697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:29.233687 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:29.233697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:29.234399 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:29.234403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:29.234642 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:29.235270 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:29.235285 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:29.237314 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:29.239680 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:29.239684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:29.239686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:29.239938 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:29.240311 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:29.240313 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:29.240314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:29.240382 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:29.241022 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:29.241025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:29.241095 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 04:39:29.241218 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 04:39:29.241661 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":147,"spinlock_wait_cycles":40448,"thread_start_us":63,"threads_started":1}
I20260924 04:39:29.242394 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:29.228136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50713 (local address 127.0.98.2:34097)
0924 04:39:29.228594 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:29.228622 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:29.228669 (+ 47us) server_negotiation.cc:407] Connection header received
0924 04:39:29.229465 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:29.229472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:29.229827 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:29.230138 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:29.231320 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:29.232531 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:29.233808 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:29.234265 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:29.234436 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:29.237128 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:29.237175 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:29.237191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:29.237224 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:29.239007 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:29.239609 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:29.239614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:29.239618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:29.239729 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:29.240097 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:29.240100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:29.240101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:29.240611 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:29.240745 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:29.241327 (+ 582us) server_negotiation.cc:299] Negotiation successful
0924 04:39:29.241611 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":128,"threads_started":1}
W20260924 04:39:29.244451 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
W20260924 04:39:29.244669 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
I20260924 04:39:29.244752 16958 leader_election.cc:302] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6cf48e1833b403bb5546aebf0181f1c; no voters: 9579d1e3079a4ad29441c5c966335347, dfac52afbf744ee0969a49bd9dde53b8
I20260924 04:39:29.245023 17343 raft_consensus.cc:2726] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:39:29.790499 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:29.790029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:43569)
0924 04:39:29.790341 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:29.790404 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
I20260924 04:39:29.897267 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:29.896799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:54055)
0924 04:39:29.897063 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:29.897145 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":68,"threads_started":1}
I20260924 04:39:30.072634 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:30.072029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:43989)
0924 04:39:30.072395 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:30.072491 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":80,"threads_started":1}
I20260924 04:39:30.791767 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:30.791390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:53535)
0924 04:39:30.791622 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:30.791674 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260924 04:39:30.898387 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:30.898001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:33539)
0924 04:39:30.898230 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:30.898280 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":68,"threads_started":1}
I20260924 04:39:31.073778 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:31.073380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:49233)
0924 04:39:31.073626 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:31.073679 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260924 04:39:31.468971 17353 raft_consensus.cc:482] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:31.469089 17353 raft_consensus.cc:504] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:31.469527 17353 leader_election.cc:288] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097), dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973)
W20260924 04:39:31.470333 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
W20260924 04:39:31.470456 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
I20260924 04:39:31.470497 16958 leader_election.cc:302] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6cf48e1833b403bb5546aebf0181f1c; no voters: 9579d1e3079a4ad29441c5c966335347, dfac52afbf744ee0969a49bd9dde53b8
I20260924 04:39:31.470630 17353 raft_consensus.cc:2726] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:39:31.792943 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:31.792527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:34361)
0924 04:39:31.792779 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:31.792843 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260924 04:39:31.899430 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:31.899050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:44941)
0924 04:39:31.899294 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:31.899345 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260924 04:39:32.074872 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:32.074490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:58231)
0924 04:39:32.074735 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:32.074788 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260924 04:39:32.794216 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:32.793786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:36363)
0924 04:39:32.794069 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:32.794123 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":71,"threads_started":1}
I20260924 04:39:32.900557 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:32.900204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:56625)
0924 04:39:32.900426 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:32.900474 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260924 04:39:33.075950 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:33.075572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:36245)
0924 04:39:33.075816 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:33.075870 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260924 04:39:33.795375 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:33.794982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:59311)
0924 04:39:33.795229 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:33.795282 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260924 04:39:33.901670 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:33.901301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:52733)
0924 04:39:33.901527 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:33.901579 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260924 04:39:34.077112 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:34.076686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:43793)
0924 04:39:34.076963 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:34.077021 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260924 04:39:34.454618 17363 raft_consensus.cc:482] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:34.454773 17363 raft_consensus.cc:504] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:34.455191 17363 leader_election.cc:288] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097), dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973)
W20260924 04:39:34.456049 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
W20260924 04:39:34.456197 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
I20260924 04:39:34.456256 16958 leader_election.cc:302] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6cf48e1833b403bb5546aebf0181f1c; no voters: 9579d1e3079a4ad29441c5c966335347, dfac52afbf744ee0969a49bd9dde53b8
I20260924 04:39:34.456441 17363 raft_consensus.cc:2726] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:39:34.796450 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:34.796043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:38659)
0924 04:39:34.796304 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:34.796357 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260924 04:39:34.902799 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:34.902373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:60469)
0924 04:39:34.902658 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:34.902707 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
I20260924 04:39:35.078439 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:35.077911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:33323)
0924 04:39:35.078285 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:35.078353 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260924 04:39:35.797657 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:35.797277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:42525)
0924 04:39:35.797513 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:35.797564 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260924 04:39:35.904019 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:35.903577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:43875)
0924 04:39:35.903879 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:35.903929 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":66,"threads_started":1}
I20260924 04:39:36.079509 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:36.079123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:39065)
0924 04:39:36.079357 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:36.079424 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260924 04:39:36.798780 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:36.798383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:40787)
0924 04:39:36.798633 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:36.798687 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260924 04:39:36.905181 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:36.904720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:57259)
0924 04:39:36.904996 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:36.905048 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":120,"threads_started":1}
I20260924 04:39:36.956413 17372 raft_consensus.cc:482] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:36.956610 17372 raft_consensus.cc:504] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:36.957010 17372 leader_election.cc:288] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097), dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973)
W20260924 04:39:36.957765 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
W20260924 04:39:36.957913 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
I20260924 04:39:36.957993 16958 leader_election.cc:302] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6cf48e1833b403bb5546aebf0181f1c; no voters: 9579d1e3079a4ad29441c5c966335347, dfac52afbf744ee0969a49bd9dde53b8
I20260924 04:39:36.958170 17372 raft_consensus.cc:2726] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:39:37.080585 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:37.080201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:46467)
0924 04:39:37.080429 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:37.080496 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260924 04:39:37.800037 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:37.799654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:38471)
0924 04:39:37.799882 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:37.799945 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260924 04:39:37.906337 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:37.905933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:51073)
0924 04:39:37.906193 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:37.906247 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
I20260924 04:39:38.081717 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:38.081342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:47309)
0924 04:39:38.081571 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:38.081627 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260924 04:39:38.801223 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:38.800783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:41657)
0924 04:39:38.801007 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:38.801060 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260924 04:39:38.907424 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:38.907012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:37045)
0924 04:39:38.907283 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:38.907336 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260924 04:39:39.082733 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:39.082350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:37005)
0924 04:39:39.082591 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:39.082647 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260924 04:39:39.802336 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:39.801963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:48877)
0924 04:39:39.802194 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:39.802245 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260924 04:39:39.908447 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:39.908075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:41841)
0924 04:39:39.908306 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:39.908358 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260924 04:39:40.083768 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:40.083381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:41983)
0924 04:39:40.083622 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:40.083676 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260924 04:39:40.803597 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:40.803088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:41059)
0924 04:39:40.803426 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:40.803498 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":90,"threads_started":1}
I20260924 04:39:40.909505 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:40.909092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:37353)
0924 04:39:40.909357 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:40.909416 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":119,"threads_started":1}
I20260924 04:39:41.084846 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:41.084451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:51249)
0924 04:39:41.084700 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:41.084753 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260924 04:39:41.804847 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:41.804399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:48357)
0924 04:39:41.804647 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:41.804699 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260924 04:39:41.910637 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:41.910234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:37923)
0924 04:39:41.910496 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:41.910547 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
I20260924 04:39:41.982555 17388 raft_consensus.cc:482] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:41.982678 17388 raft_consensus.cc:504] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9579d1e3079a4ad29441c5c966335347" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34097 } } peers { permanent_uuid: "dfac52afbf744ee0969a49bd9dde53b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38973 } } peers { permanent_uuid: "b6cf48e1833b403bb5546aebf0181f1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40195 } }
I20260924 04:39:41.983008 17388 leader_election.cc:288] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097), dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973)
W20260924 04:39:41.984031 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9579d1e3079a4ad29441c5c966335347 (127.0.98.2:34097): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
W20260924 04:39:41.984186 16958 leader_election.cc:341] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfac52afbf744ee0969a49bd9dde53b8 (127.0.98.3:38973): Not found: Tablet not found: a1953c4fb8ef46328653096b8311ae33
I20260924 04:39:41.984251 16958 leader_election.cc:302] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6cf48e1833b403bb5546aebf0181f1c; no voters: 9579d1e3079a4ad29441c5c966335347, dfac52afbf744ee0969a49bd9dde53b8
I20260924 04:39:41.984442 17388 raft_consensus.cc:2726] T a1953c4fb8ef46328653096b8311ae33 P b6cf48e1833b403bb5546aebf0181f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:39:42.085948 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:42.085538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:37885)
0924 04:39:42.085775 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:42.085827 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260924 04:39:42.806210 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:42.805809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.1:41363)
0924 04:39:42.806066 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:42.806118 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260924 04:39:42.911746 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:42.911383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.2:45617)
0924 04:39:42.911614 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:42.911661 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
I20260924 04:39:43.077275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 16900
I20260924 04:39:43.083051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 17072
I20260924 04:39:43.087198 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:43.086821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36685 (local address 127.0.98.3:47451)
0924 04:39:43.087062 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:43.087114 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260924 04:39:43.088281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 17207
2026-09-24T04:39:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59314 ms)
[----------] 30 tests from SecurityITest (237782 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:39:43 dist-test-slave-jlnt krb5kdc[17395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:39:43 dist-test-slave-jlnt krb5kdc[17395](info): set up 2 sockets
Sep 24 04:39:43 dist-test-slave-jlnt krb5kdc[17395](info): commencing operation
krb5kdc: starting...
W20260924 04:39:44.136747 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:39:44 dist-test-slave-jlnt krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:39:44.253333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37867
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:44.361303 17408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:44.361524 17408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:44.361562 17408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:44.365427 17408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:44.365471 17408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:44.365490 17408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:44.365506 17408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:44.365521 17408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:44.370555 17408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:44.371397 17408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:44.372283 17408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:44.380462 17416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:44.380566 17408 server_base.cc:1034] running on GCE node
W20260924 04:39:44.380512 17413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:44.380512 17414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:44.381309 17408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:44.382280 17408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:44.383432 17408 hybrid_clock.cc:648] HybridClock initialized: now 1790224784383414 us; error 29 us; skew 500 ppm
Sep 24 04:39:44 dist-test-slave-jlnt krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:44.386096 17408 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:39:44.386879 17408 webserver.cc:460] Webserver started at http://127.0.98.62:45629/ using document root <none> and password file <none>
I20260924 04:39:44.387395 17408 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:44.387454 17408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:44.387707 17408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:44.389647 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "899b51b7679747b4baa5f2482069dcc6"
format_stamp: "Formatted at 2026-09-24 04:39:44 on dist-test-slave-jlnt"
server_key: "801c8c22027329fa7c659cf757eadf98"
server_key_iv: "b6bd34d58fc77af66426a582522731b8"
server_key_version: "encryptionkey@0"
I20260924 04:39:44.390143 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "899b51b7679747b4baa5f2482069dcc6"
format_stamp: "Formatted at 2026-09-24 04:39:44 on dist-test-slave-jlnt"
server_key: "801c8c22027329fa7c659cf757eadf98"
server_key_iv: "b6bd34d58fc77af66426a582522731b8"
server_key_version: "encryptionkey@0"
I20260924 04:39:44.393971 17408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:39:44.396118 17423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:44.397423 17408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:39:44.397547 17408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "899b51b7679747b4baa5f2482069dcc6"
format_stamp: "Formatted at 2026-09-24 04:39:44 on dist-test-slave-jlnt"
server_key: "801c8c22027329fa7c659cf757eadf98"
server_key_iv: "b6bd34d58fc77af66426a582522731b8"
server_key_version: "encryptionkey@0"
I20260924 04:39:44.397646 17408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:44.418342 17408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:44.421283 17408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:44.421480 17408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:44.428143 17408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37867
I20260924 04:39:44.428169 17475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37867 every 8 connection(s)
I20260924 04:39:44.429026 17408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:44.431629 17476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:44.436697 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6: Bootstrap starting.
I20260924 04:39:44.438638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 17408
I20260924 04:39:44.438808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:44.439061 17476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:44.439087 392 external_mini_cluster.cc:1428] Setting key aa36a608285903d0564fb6dd7dc0f5b2
I20260924 04:39:44.439738 17476 log.cc:824] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:44.441913 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6: No bootstrap required, opened a new log
I20260924 04:39:44.446123 17476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899b51b7679747b4baa5f2482069dcc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37867 } }
I20260924 04:39:44.446408 17476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:44.446480 17476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899b51b7679747b4baa5f2482069dcc6, State: Initialized, Role: FOLLOWER
I20260924 04:39:44.446949 17476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899b51b7679747b4baa5f2482069dcc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37867 } }
I20260924 04:39:44.447103 17476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:44.447171 17476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:44.447268 17476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:44.448263 17476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899b51b7679747b4baa5f2482069dcc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37867 } }
Sep 24 04:39:44 dist-test-slave-jlnt krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:44.448637 17476 leader_election.cc:302] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 899b51b7679747b4baa5f2482069dcc6; no voters:
I20260924 04:39:44.448927 17476 leader_election.cc:288] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:44.449038 17481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:44.449287 17481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [term 1 LEADER]: Becoming Leader. State: Replica: 899b51b7679747b4baa5f2482069dcc6, State: Running, Role: LEADER
I20260924 04:39:44.449656 17481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899b51b7679747b4baa5f2482069dcc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37867 } }
I20260924 04:39:44.450263 17476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:44.451277 17482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899b51b7679747b4baa5f2482069dcc6. Latest consensus state: current_term: 1 leader_uuid: "899b51b7679747b4baa5f2482069dcc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899b51b7679747b4baa5f2482069dcc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37867 } } }
I20260924 04:39:44.451504 17482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:44.451644 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "899b51b7679747b4baa5f2482069dcc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899b51b7679747b4baa5f2482069dcc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37867 } } }
I20260924 04:39:44.451814 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:44.451960 17490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:44.454820 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:44.440109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.0.98.62:37867)
0924 04:39:44.440577 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:44.440597 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:44.440639 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:39:44.441472 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:44.441494 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:44.441865 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:44.442213 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:44.442907 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.444407 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.445339 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.446024 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.446187 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:44.449193 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:44.449199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:44.449214 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:44.449244 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:44.451402 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:44.451948 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:44.451954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:44.451960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:44.452097 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:44.452437 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:44.452441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:44.452446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:44.453002 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:44.453218 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:44.453619 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 04:39:44.453930 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260924 04:39:44.455031 17490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:44.461117 17490 catalog_manager.cc:1347] Generated new cluster ID: d891f8cd663247b4b420fb66459d6540
I20260924 04:39:44.461210 17490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:44.472153 17490 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:44.473018 17490 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:44.477037 17490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6: Generated new TSK 0
I20260924 04:39:44.477710 17490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:39:44.511984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37867
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:44.624688 17502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:44.624954 17502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:44.625029 17502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:44.629020 17502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:44.629096 17502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:44.629204 17502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:39:44.634169 17502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:44.635088 17502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:44.636041 17502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:44.645010 17508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:44.644999 17507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:44.644999 17510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:44.645450 17502 server_base.cc:1034] running on GCE node
I20260924 04:39:44.645800 17502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:44.646337 17502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:44.647501 17502 hybrid_clock.cc:648] HybridClock initialized: now 1790224784647487 us; error 26 us; skew 500 ppm
Sep 24 04:39:44 dist-test-slave-jlnt krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:44.650382 17502 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:39:44.651319 17502 webserver.cc:460] Webserver started at http://127.0.98.1:38085/ using document root <none> and password file <none>
I20260924 04:39:44.651850 17502 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:44.651909 17502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:44.652124 17502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:44.654153 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "42508ea8743a4ab39f7d2dd576e96407"
format_stamp: "Formatted at 2026-09-24 04:39:44 on dist-test-slave-jlnt"
server_key: "8623c4e67aece5697b7b8fdada60a24d"
server_key_iv: "4148a89a53037a6957e0223600d2184a"
server_key_version: "encryptionkey@0"
I20260924 04:39:44.654680 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "42508ea8743a4ab39f7d2dd576e96407"
format_stamp: "Formatted at 2026-09-24 04:39:44 on dist-test-slave-jlnt"
server_key: "8623c4e67aece5697b7b8fdada60a24d"
server_key_iv: "4148a89a53037a6957e0223600d2184a"
server_key_version: "encryptionkey@0"
I20260924 04:39:44.658088 17502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:39:44.660405 17517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:44.661588 17502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:44.661711 17502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "42508ea8743a4ab39f7d2dd576e96407"
format_stamp: "Formatted at 2026-09-24 04:39:44 on dist-test-slave-jlnt"
server_key: "8623c4e67aece5697b7b8fdada60a24d"
server_key_iv: "4148a89a53037a6957e0223600d2184a"
server_key_version: "encryptionkey@0"
I20260924 04:39:44.661815 17502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:44.676826 17502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:44.679741 17502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:44.679980 17502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:44.680505 17502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:44.681269 17502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:44.681318 17502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:44.681391 17502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:44.681422 17502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:44.689422 17502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43025
I20260924 04:39:44.689508 17630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43025 every 8 connection(s)
I20260924 04:39:44.690253 17502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:39:44.697949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 17502
I20260924 04:39:44.698048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:39:44.698360 392 external_mini_cluster.cc:1428] Setting key ac09eecc50c6cf435151a5f0f04a8867
Sep 24 04:39:44 dist-test-slave-jlnt krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:44.703200 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:44.691887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54161 (local address 127.0.98.62:37867)
0924 04:39:44.692030 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:44.692034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:44.692706 (+ 672us) server_negotiation.cc:407] Connection header received
0924 04:39:44.693663 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:44.693666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:44.693722 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:44.693831 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:44.694876 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.695400 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.696491 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.696736 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.696849 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:44.699313 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:44.699317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:44.699320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:44.699352 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:44.700903 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:44.701529 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:44.701532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:44.701533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:44.701609 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:44.701997 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:44.702000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:44.702001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:44.702160 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:44.702233 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:44.702803 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 04:39:44.702960 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:39:44.703665 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:44.692148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37867 (local address 127.0.98.1:54161)
0924 04:39:44.692565 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:44.692603 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:44.693467 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:44.693990 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:44.693997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:44.694325 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:44.694737 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:44.694749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.695523 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.695531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:44.696365 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:44.696374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.696883 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.696887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:44.697044 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:44.697615 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:44.697630 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:44.699144 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:44.701016 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:44.701021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:44.701034 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:44.701433 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:44.701714 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:44.701719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:44.701722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:44.701897 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:44.702263 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:44.702270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:44.702562 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 04:39:44.702703 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 04:39:44.702920 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260924 04:39:44.704914 17631 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37867
I20260924 04:39:44.705179 17631 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:44.705605 17631 heartbeater.cc:511] Master 127.0.98.62:37867 requested a full tablet report, sending...
I20260924 04:39:44.707259 17440 ts_manager.cc:194] Registered new tserver with Master: 42508ea8743a4ab39f7d2dd576e96407 (127.0.98.1:43025)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:39:44.708791 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:39:44.725641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37867
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:44.830874 17636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:44.831132 17636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:44.831207 17636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:44.835168 17636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:44.835244 17636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:44.835330 17636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:39:44.840229 17636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:44.841187 17636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:44.842104 17636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:44.851033 17641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:44.851094 17642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:44.851096 17644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:44.851220 17636 server_base.cc:1034] running on GCE node
I20260924 04:39:44.851814 17636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:44.852433 17636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:44.853602 17636 hybrid_clock.cc:648] HybridClock initialized: now 1790224784853591 us; error 25 us; skew 500 ppm
Sep 24 04:39:44 dist-test-slave-jlnt krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:44.856114 17636 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:39:44.856905 17636 webserver.cc:460] Webserver started at http://127.0.98.2:45681/ using document root <none> and password file <none>
I20260924 04:39:44.857481 17636 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:44.857568 17636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:44.857776 17636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:44.859604 17636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "305691a24e7c4426ae67f5fd3e2c75af"
format_stamp: "Formatted at 2026-09-24 04:39:44 on dist-test-slave-jlnt"
server_key: "4f3b3b9a6e3a2ab02deafb4f955b1378"
server_key_iv: "1a2aba0b9855ab8f89253dd2b920e26d"
server_key_version: "encryptionkey@0"
I20260924 04:39:44.860191 17636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "305691a24e7c4426ae67f5fd3e2c75af"
format_stamp: "Formatted at 2026-09-24 04:39:44 on dist-test-slave-jlnt"
server_key: "4f3b3b9a6e3a2ab02deafb4f955b1378"
server_key_iv: "1a2aba0b9855ab8f89253dd2b920e26d"
server_key_version: "encryptionkey@0"
I20260924 04:39:44.863647 17636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:39:44.866024 17651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:44.867167 17636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:44.867303 17636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "305691a24e7c4426ae67f5fd3e2c75af"
format_stamp: "Formatted at 2026-09-24 04:39:44 on dist-test-slave-jlnt"
server_key: "4f3b3b9a6e3a2ab02deafb4f955b1378"
server_key_iv: "1a2aba0b9855ab8f89253dd2b920e26d"
server_key_version: "encryptionkey@0"
I20260924 04:39:44.867406 17636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:44.908773 17636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:44.911757 17636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:44.911954 17636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:44.912508 17636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:44.913292 17636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:44.913363 17636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:44.913431 17636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:44.913480 17636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:44.921940 17636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42681
I20260924 04:39:44.922010 17764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42681 every 8 connection(s)
I20260924 04:39:44.923044 17636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:39:44.931214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 17636
I20260924 04:39:44.931313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:39:44.931591 392 external_mini_cluster.cc:1428] Setting key 651111b04410009a07c0d165bf713952
Sep 24 04:39:44 dist-test-slave-jlnt krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:44.936164 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:44.924815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40453 (local address 127.0.98.62:37867)
0924 04:39:44.924957 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:44.924962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:44.925996 (+ 1034us) server_negotiation.cc:407] Connection header received
0924 04:39:44.926794 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:44.926796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:44.926840 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:44.926910 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:44.928161 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.928675 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.929833 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.930062 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.930145 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:44.932574 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:44.932580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:44.932587 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:44.932610 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:44.934119 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:44.934759 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:44.934762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:44.934763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:44.934840 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:44.935205 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:44.935207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:44.935208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:44.935365 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:44.935438 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:44.935953 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 04:39:44.936049 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:39:44.936909 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:44.925206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37867 (local address 127.0.98.2:40453)
0924 04:39:44.925853 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:44.925891 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:44.926622 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:44.927030 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:44.927036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:44.927373 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:44.928006 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:44.928024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.928787 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.928815 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:44.929716 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:44.929724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:44.930186 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:44.930188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:44.930325 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:44.930816 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:44.930831 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:44.932411 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:44.934250 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:44.934256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:44.934268 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:44.934641 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:44.934931 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:44.934935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:44.934939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:44.935114 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:44.935466 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:44.935474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:44.935756 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 04:39:44.935912 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 04:39:44.936130 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":110,"threads_started":1}
I20260924 04:39:44.938186 17765 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37867
I20260924 04:39:44.938432 17765 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:44.938987 17765 heartbeater.cc:511] Master 127.0.98.62:37867 requested a full tablet report, sending...
I20260924 04:39:44.940124 17440 ts_manager.cc:194] Registered new tserver with Master: 305691a24e7c4426ae67f5fd3e2c75af (127.0.98.2:42681)
I20260924 04:39:44.940662 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40453
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:39:44.959414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37867
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:45.066192 17770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:45.066409 17770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:45.066479 17770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:45.070529 17770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:45.070581 17770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:45.070683 17770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:39:45.075738 17770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:45.076686 17770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:45.077710 17770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:45.087316 17778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:45.087468 17770 server_base.cc:1034] running on GCE node
W20260924 04:39:45.087361 17775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:45.087328 17776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:45.087945 17770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:45.088483 17770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:45.089637 17770 hybrid_clock.cc:648] HybridClock initialized: now 1790224785089622 us; error 25 us; skew 500 ppm
Sep 24 04:39:45 dist-test-slave-jlnt krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:45.092343 17770 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:39:45.093210 17770 webserver.cc:460] Webserver started at http://127.0.98.3:40329/ using document root <none> and password file <none>
I20260924 04:39:45.093739 17770 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:45.093801 17770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:45.094046 17770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:45.095922 17770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "43fd6c2afb76b73fd3631d213cb45b1c"
server_key_iv: "e983fd91727acd512ecff090ff34b5ff"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.096655 17770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "43fd6c2afb76b73fd3631d213cb45b1c"
server_key_iv: "e983fd91727acd512ecff090ff34b5ff"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.100225 17770 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:39:45.102656 17785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.103807 17770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:39:45.103931 17770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "43fd6c2afb76b73fd3631d213cb45b1c"
server_key_iv: "e983fd91727acd512ecff090ff34b5ff"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.104029 17770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:45.117262 17770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:45.120297 17770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:45.120448 17770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:45.121202 17770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:45.121945 17770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:45.121991 17770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.122057 17770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:45.122088 17770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.130491 17770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45705
I20260924 04:39:45.130548 17898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45705 every 8 connection(s)
I20260924 04:39:45.131295 17770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:39:45.134639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 17770
I20260924 04:39:45.134775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:39:45.135077 392 external_mini_cluster.cc:1428] Setting key 69d74600d15c9d15f949370b169e7136
Sep 24 04:39:45 dist-test-slave-jlnt krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:45.143900 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.133034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50281 (local address 127.0.98.62:37867)
0924 04:39:45.133213 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.133217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.133985 (+ 768us) server_negotiation.cc:407] Connection header received
0924 04:39:45.135004 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.135007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.135056 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.135124 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:45.136533 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.137063 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.138203 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.138427 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.138545 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.140724 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:45.140730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:45.140737 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:45.140760 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:45.142017 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:45.142492 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:45.142496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:45.142498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:45.142575 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:45.142897 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:45.142901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:45.142903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:45.143073 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:45.143180 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.143621 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.143730 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:39:45.144629 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.133433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37867 (local address 127.0.98.3:50281)
0924 04:39:45.133834 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.133874 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:45.134761 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:45.135321 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:45.135329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:45.135795 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:45.136376 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.136395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.137217 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.137220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.138078 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.138087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.138577 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.138580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.138746 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.139243 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:45.139258 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:45.140573 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:45.142133 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:45.142138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:45.142151 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:45.142386 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:45.142667 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:45.142671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:45.142673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:45.142801 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:45.143178 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:45.143185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:45.143402 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 04:39:45.143580 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 04:39:45.143860 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":101,"threads_started":1}
I20260924 04:39:45.145681 17899 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37867
I20260924 04:39:45.145895 17899 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:45.146335 17899 heartbeater.cc:511] Master 127.0.98.62:37867 requested a full tablet report, sending...
I20260924 04:39:45.147286 17440 ts_manager.cc:194] Registered new tserver with Master: 4b06e292833b4f79aa69ef7c6c2cdcc2 (127.0.98.3:45705)
I20260924 04:39:45.147881 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50281
I20260924 04:39:45.148586 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:39:45.157003 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.150527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.0.98.62:37867)
0924 04:39:45.150628 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.150631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.150814 (+ 183us) server_negotiation.cc:407] Connection header received
0924 04:39:45.150892 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.150894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.150947 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.151044 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:45.151640 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.152155 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.152967 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.153213 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.153287 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.154494 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:45.154497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:45.154498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:45.154514 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:45.155684 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:45.156076 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:45.156078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:45.156079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:45.156143 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:45.156354 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:45.156356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:45.156357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:45.156543 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:45.156643 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.156826 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.156907 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:39:45.159651 17440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:39:45.162654 17440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:39:45.178412 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.170458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45705 (local address 127.0.0.1:51290)
0924 04:39:45.171092 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.171143 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:45.171439 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:45.172692 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:45.172698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:45.172720 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:45.172855 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.172862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.174276 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.174287 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.175957 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.175972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.177209 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.177212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.177982 (+ 770us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.178031 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 04:39:45.178146 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:39:45.178266 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":552,"thread_start_us":66,"threads_started":1}
I20260924 04:39:45.178418 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.171085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.0.98.3:45705)
0924 04:39:45.172182 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.172196 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.172221 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:39:45.172295 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.172301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.172506 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.173013 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:45.173208 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.174133 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.176093 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.176904 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.176995 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.177126 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.178176 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.178275 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1003,"thread_start_us":141,"threads_started":1}
I20260924 04:39:45.179548 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.172140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43025 (local address 127.0.0.1:40028)
0924 04:39:45.174635 (+ 2495us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.174648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:45.174734 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:45.175379 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:45.175382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:45.175398 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:45.175472 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.175477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.176858 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.176862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.178106 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.178113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.179138 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.179142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.179239 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.179262 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:39:45.179349 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:39:45.179424 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2433,"mutex_wait_us":70,"thread_start_us":100,"threads_started":1}
I20260924 04:39:45.179636 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.172520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.0.98.1:43025)
0924 04:39:45.173006 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.173022 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.174838 (+ 1816us) server_negotiation.cc:407] Connection header received
0924 04:39:45.174915 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.174920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.175088 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.175228 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:45.175585 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.176698 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.178224 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.179013 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.179127 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.179266 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.179379 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.179473 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":93,"threads_started":1}
I20260924 04:39:45.180130 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.170608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42681 (local address 127.0.0.1:60062)
0924 04:39:45.172475 (+ 1867us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.172491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:45.172613 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:45.174970 (+ 2357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:45.174973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:45.174990 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:45.175067 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.175073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.176450 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.176453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.177894 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.177904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.179474 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.179478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.179570 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.179591 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:39:45.179898 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 04:39:45.180012 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1740,"mutex_wait_us":232,"thread_start_us":83,"threads_started":1}
I20260924 04:39:45.180622 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.171565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.0.98.2:42681)
0924 04:39:45.174373 (+ 2808us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.174381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.174398 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:39:45.174453 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.174456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.174662 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.174802 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:45.175196 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.176288 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.178022 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.179183 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.179273 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.179416 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.180405 (+ 989us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.180503 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2726,"thread_start_us":100,"threads_started":1}
I20260924 04:39:45.181581 17833 tablet_service.cc:1467] Processing CreateTablet for tablet 5a234dcc9caf4517b4708a2542cb2a20 (DEFAULT_TABLE table=test-table [id=b96005e629114aa5a6e908d52dc2d4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:45.182687 17833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a234dcc9caf4517b4708a2542cb2a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:45.182386 17565 tablet_service.cc:1467] Processing CreateTablet for tablet 5a234dcc9caf4517b4708a2542cb2a20 (DEFAULT_TABLE table=test-table [id=b96005e629114aa5a6e908d52dc2d4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:45.182933 17699 tablet_service.cc:1467] Processing CreateTablet for tablet 5a234dcc9caf4517b4708a2542cb2a20 (DEFAULT_TABLE table=test-table [id=b96005e629114aa5a6e908d52dc2d4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:45.183359 17565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a234dcc9caf4517b4708a2542cb2a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:45.183727 17699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a234dcc9caf4517b4708a2542cb2a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:45.189234 17916 tablet_bootstrap.cc:492] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: Bootstrap starting.
I20260924 04:39:45.190464 17918 tablet_bootstrap.cc:492] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407: Bootstrap starting.
I20260924 04:39:45.190924 17917 tablet_bootstrap.cc:492] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: Bootstrap starting.
I20260924 04:39:45.192122 17916 tablet_bootstrap.cc:654] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:45.192281 17918 tablet_bootstrap.cc:654] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:45.192910 17917 tablet_bootstrap.cc:654] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:45.193365 17918 log.cc:824] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:45.193432 17916 log.cc:824] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:45.193985 17917 log.cc:824] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:45.195178 17918 tablet_bootstrap.cc:492] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407: No bootstrap required, opened a new log
I20260924 04:39:45.195247 17916 tablet_bootstrap.cc:492] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: No bootstrap required, opened a new log
I20260924 04:39:45.195391 17918 ts_tablet_manager.cc:1397] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 04:39:45.195596 17916 ts_tablet_manager.cc:1397] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:39:45.195781 17917 tablet_bootstrap.cc:492] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: No bootstrap required, opened a new log
I20260924 04:39:45.196204 17917 ts_tablet_manager.cc:1397] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:39:45.198437 17918 raft_consensus.cc:348] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } }
I20260924 04:39:45.198664 17918 raft_consensus.cc:374] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:45.198737 17918 raft_consensus.cc:729] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42508ea8743a4ab39f7d2dd576e96407, State: Initialized, Role: FOLLOWER
I20260924 04:39:45.199172 17918 consensus_queue.cc:260] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } }
I20260924 04:39:45.199463 17916 raft_consensus.cc:348] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } }
I20260924 04:39:45.199774 17631 heartbeater.cc:503] Master 127.0.98.62:37867 was elected leader, sending a full tablet report...
I20260924 04:39:45.199764 17916 raft_consensus.cc:374] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:45.199785 17917 raft_consensus.cc:348] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } }
I20260924 04:39:45.199983 17916 raft_consensus.cc:729] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305691a24e7c4426ae67f5fd3e2c75af, State: Initialized, Role: FOLLOWER
I20260924 04:39:45.200060 17917 raft_consensus.cc:374] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:45.200098 17918 ts_tablet_manager.cc:1428] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:39:45.200148 17917 raft_consensus.cc:729] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b06e292833b4f79aa69ef7c6c2cdcc2, State: Initialized, Role: FOLLOWER
I20260924 04:39:45.200560 17917 consensus_queue.cc:260] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } }
I20260924 04:39:45.200569 17916 consensus_queue.cc:260] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } }
I20260924 04:39:45.201200 17765 heartbeater.cc:503] Master 127.0.98.62:37867 was elected leader, sending a full tablet report...
I20260924 04:39:45.201475 17899 heartbeater.cc:503] Master 127.0.98.62:37867 was elected leader, sending a full tablet report...
I20260924 04:39:45.201591 17916 ts_tablet_manager.cc:1428] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:39:45.202580 17917 ts_tablet_manager.cc:1428] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:39:45.272861 17923 raft_consensus.cc:482] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:45.273088 17923 raft_consensus.cc:504] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } }
I20260924 04:39:45.274232 17923 leader_election.cc:288] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b06e292833b4f79aa69ef7c6c2cdcc2 (127.0.98.3:45705), 42508ea8743a4ab39f7d2dd576e96407 (127.0.98.1:43025)
I20260924 04:39:45.278086 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.274494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43025 (local address 127.0.98.2:57099)
0924 04:39:45.274855 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.274869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:45.274985 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:45.275265 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:45.275267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:45.275281 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:45.275414 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.275421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.276143 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.276145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.276954 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.276971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.277634 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.277640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.277740 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.277771 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:39:45.277850 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:39:45.277903 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":4,"thread_start_us":139,"threads_started":1}
I20260924 04:39:45.278092 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.274577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57099 (local address 127.0.98.1:43025)
0924 04:39:45.274690 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.274695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.274987 (+ 292us) server_negotiation.cc:407] Connection header received
0924 04:39:45.275110 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.275113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.275161 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.275249 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:45.275529 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.276001 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.277091 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.277514 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.277595 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.277644 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.277880 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.277971 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:39:45.278208 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.274723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57115 (local address 127.0.98.3:45705)
0924 04:39:45.274839 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.274843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.274857 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:39:45.274918 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.274922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.274976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.275052 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:45.275401 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.275971 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.277126 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.277604 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.277707 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.277771 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.278002 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.278075 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:39:45.278138 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.274472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45705 (local address 127.0.98.2:57115)
0924 04:39:45.274667 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.274682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:45.274779 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:45.275106 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:45.275109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:45.275129 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:45.275272 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.275279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.276091 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.276097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.276977 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.276986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.277738 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.277741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.277866 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.277889 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:39:45.277965 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:39:45.278027 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":50}
I20260924 04:39:45.278872 17585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a234dcc9caf4517b4708a2542cb2a20" candidate_uuid: "305691a24e7c4426ae67f5fd3e2c75af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42508ea8743a4ab39f7d2dd576e96407" is_pre_election: true
I20260924 04:39:45.279007 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a234dcc9caf4517b4708a2542cb2a20" candidate_uuid: "305691a24e7c4426ae67f5fd3e2c75af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" is_pre_election: true
I20260924 04:39:45.279163 17585 raft_consensus.cc:2454] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 305691a24e7c4426ae67f5fd3e2c75af in term 0.
I20260924 04:39:45.279238 17853 raft_consensus.cc:2454] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 305691a24e7c4426ae67f5fd3e2c75af in term 0.
I20260924 04:39:45.279698 17655 leader_election.cc:302] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 305691a24e7c4426ae67f5fd3e2c75af, 42508ea8743a4ab39f7d2dd576e96407, 4b06e292833b4f79aa69ef7c6c2cdcc2; no voters:
I20260924 04:39:45.279955 17923 raft_consensus.cc:2781] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:39:45.280035 17923 raft_consensus.cc:482] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:39:45.280066 17923 raft_consensus.cc:3037] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:45.281080 17923 raft_consensus.cc:504] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } }
I20260924 04:39:45.281426 17923 leader_election.cc:288] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [CANDIDATE]: Term 1 election: Requested vote from peers 4b06e292833b4f79aa69ef7c6c2cdcc2 (127.0.98.3:45705), 42508ea8743a4ab39f7d2dd576e96407 (127.0.98.1:43025)
I20260924 04:39:45.281807 17585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a234dcc9caf4517b4708a2542cb2a20" candidate_uuid: "305691a24e7c4426ae67f5fd3e2c75af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42508ea8743a4ab39f7d2dd576e96407"
I20260924 04:39:45.281806 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a234dcc9caf4517b4708a2542cb2a20" candidate_uuid: "305691a24e7c4426ae67f5fd3e2c75af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2"
I20260924 04:39:45.281930 17585 raft_consensus.cc:3037] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:45.281930 17853 raft_consensus.cc:3037] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:45.283198 17853 raft_consensus.cc:2454] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305691a24e7c4426ae67f5fd3e2c75af in term 1.
I20260924 04:39:45.283198 17585 raft_consensus.cc:2454] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305691a24e7c4426ae67f5fd3e2c75af in term 1.
I20260924 04:39:45.283526 17654 leader_election.cc:302] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305691a24e7c4426ae67f5fd3e2c75af, 42508ea8743a4ab39f7d2dd576e96407; no voters:
I20260924 04:39:45.283716 17923 raft_consensus.cc:2781] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:45.283942 17923 raft_consensus.cc:686] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 1 LEADER]: Becoming Leader. State: Replica: 305691a24e7c4426ae67f5fd3e2c75af, State: Running, Role: LEADER
I20260924 04:39:45.284243 17923 consensus_queue.cc:237] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } }
I20260924 04:39:45.287112 17438 catalog_manager.cc:5495] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af reported cstate change: term changed from 0 to 1, leader changed from <none> to 305691a24e7c4426ae67f5fd3e2c75af (127.0.98.2). New cstate: current_term: 1 leader_uuid: "305691a24e7c4426ae67f5fd3e2c75af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "305691a24e7c4426ae67f5fd3e2c75af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } health_report { overall_health: UNKNOWN } } }
I20260924 04:39:45.300201 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.296680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.0.98.2:42681)
0924 04:39:45.296793 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.296798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.296865 (+ 67us) server_negotiation.cc:407] Connection header received
0924 04:39:45.297042 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.297047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.297097 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.297255 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:39:45.297510 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.297985 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.298836 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.299059 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.299168 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.299453 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:39:45.299838 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:39:45.299976 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.300012 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.300059 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:39:45.305485 17853 raft_consensus.cc:1261] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [term 1 FOLLOWER]: Refusing update from remote peer 305691a24e7c4426ae67f5fd3e2c75af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:45.305486 17585 raft_consensus.cc:1261] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [term 1 FOLLOWER]: Refusing update from remote peer 305691a24e7c4426ae67f5fd3e2c75af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:45.306107 17926 consensus_queue.cc:1035] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [LEADER]: Connected to new peer: Peer: permanent_uuid: "42508ea8743a4ab39f7d2dd576e96407" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:45.306245 17923 consensus_queue.cc:1035] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b06e292833b4f79aa69ef7c6c2cdcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:45.312569 17929 mvcc.cc:205] Tried to move back new op lower bound from 7332760720605061120 to 7332760720526598144. Current Snapshot: MvccSnapshot[applied={T|T < 7332760720605061120}]
I20260924 04:39:45.314937 17930 mvcc.cc:205] Tried to move back new op lower bound from 7332760720605061120 to 7332760720526598144. Current Snapshot: MvccSnapshot[applied={T|T < 7332760720605061120}]
I20260924 04:39:45.322286 17439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:45.322412 17439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:45.324726 17439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 899b51b7679747b4baa5f2482069dcc6: Sending DeleteTablet for 3 replicas of tablet 5a234dcc9caf4517b4708a2542cb2a20
I20260924 04:39:45.325482 17833 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a234dcc9caf4517b4708a2542cb2a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51290
I20260924 04:39:45.325480 17699 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a234dcc9caf4517b4708a2542cb2a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60062
I20260924 04:39:45.325515 17565 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a234dcc9caf4517b4708a2542cb2a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40028
I20260924 04:39:45.325933 17940 tablet_replica.cc:330] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407: stopping tablet replica
I20260924 04:39:45.326148 17939 tablet_replica.cc:330] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: stopping tablet replica
I20260924 04:39:45.326249 17940 raft_consensus.cc:2229] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:45.326336 17938 tablet_replica.cc:330] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: stopping tablet replica
I20260924 04:39:45.326609 17940 raft_consensus.cc:2258] T 5a234dcc9caf4517b4708a2542cb2a20 P 42508ea8743a4ab39f7d2dd576e96407 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:45.326644 17938 raft_consensus.cc:2229] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:45.326665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 17502
I20260924 04:39:45.329411 17939 raft_consensus.cc:2229] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:39:45.329762 17939 raft_consensus.cc:2258] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:45.330174 17938 raft_consensus.cc:2258] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:45.331236 17939 ts_tablet_manager.cc:1905] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:45.331573 17938 ts_tablet_manager.cc:1905] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:45.334395 17938 ts_tablet_manager.cc:1918] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:45.334558 17938 log.cc:1196] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/5a234dcc9caf4517b4708a2542cb2a20
W20260924 04:39:45.334548 17426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43025 (error 108)
I20260924 04:39:45.334640 17939 ts_tablet_manager.cc:1918] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:45.334712 17939 log.cc:1196] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/5a234dcc9caf4517b4708a2542cb2a20
I20260924 04:39:45.334934 17938 ts_tablet_manager.cc:1939] T 5a234dcc9caf4517b4708a2542cb2a20 P 4b06e292833b4f79aa69ef7c6c2cdcc2: Deleting consensus metadata
I20260924 04:39:45.335045 17939 ts_tablet_manager.cc:1939] T 5a234dcc9caf4517b4708a2542cb2a20 P 305691a24e7c4426ae67f5fd3e2c75af: Deleting consensus metadata
I20260924 04:39:45.335484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 17636
I20260924 04:39:45.336057 17427 catalog_manager.cc:4841] TS 4b06e292833b4f79aa69ef7c6c2cdcc2 (127.0.98.3:45705): tablet 5a234dcc9caf4517b4708a2542cb2a20 (table test-table [id=b96005e629114aa5a6e908d52dc2d4a1]) successfully deleted
I20260924 04:39:45.337836 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.337575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43025 (local address 127.0.0.1:40036)
0924 04:39:45.337699 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.337765 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:39:45.337984 17426 catalog_manager.cc:4584] TS 42508ea8743a4ab39f7d2dd576e96407 (127.0.98.1:43025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a234dcc9caf4517b4708a2542cb2a20: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43025: connect: Connection refused (error 111)
I20260924 04:39:45.341501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 17770
I20260924 04:39:45.341960 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.341674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42681 (local address 127.0.0.1:60078)
0924 04:39:45.341835 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.341884 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 04:39:45.347148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 17408
2026-09-24T04:39:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2259 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T04:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:45Z Disabled control of system clock
I20260924 04:39:45.377334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:39:45.481933 17945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:45.482153 17945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:45.482187 17945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:45.486045 17945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:45.486092 17945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:45.486107 17945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:45.486123 17945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:45.486138 17945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:45.490607 17945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:45.491454 17945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:45.492714 17945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:45.500720 17950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:45.500757 17953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:45.500746 17951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:45.500821 17945 server_base.cc:1034] running on GCE node
I20260924 04:39:45.501528 17945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:45.502560 17945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:45.503748 17945 hybrid_clock.cc:648] HybridClock initialized: now 1790224785503736 us; error 36 us; skew 500 ppm
I20260924 04:39:45.505605 17945 webserver.cc:460] Webserver started at http://127.0.98.62:40207/ using document root <none> and password file <none>
I20260924 04:39:45.506135 17945 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:45.506227 17945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:45.506453 17945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:45.508307 17945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "0b56116dd2574110ac13d32ed4851c2d"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "7345fa85fe5545010e74fd938b5e9d5c"
server_key_iv: "4360544dd104a033536ccb91a63939c1"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.508860 17945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b56116dd2574110ac13d32ed4851c2d"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "7345fa85fe5545010e74fd938b5e9d5c"
server_key_iv: "4360544dd104a033536ccb91a63939c1"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.512341 17945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:39:45.514582 17959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.515713 17945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:39:45.515847 17945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "0b56116dd2574110ac13d32ed4851c2d"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "7345fa85fe5545010e74fd938b5e9d5c"
server_key_iv: "4360544dd104a033536ccb91a63939c1"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.515951 17945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:45.533689 17945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:45.534273 17945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:45.534462 17945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:45.540843 17945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34029
I20260924 04:39:45.540848 18011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34029 every 8 connection(s)
I20260924 04:39:45.541754 17945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:45.542514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 17945
I20260924 04:39:45.542644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:45.542930 392 external_mini_cluster.cc:1428] Setting key 596fd0afd47f6f2b245ed7b9a174b776
I20260924 04:39:45.545208 18012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:45.551172 18012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d: Bootstrap starting.
I20260924 04:39:45.553305 18012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:45.554020 18012 log.cc:824] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:45.555783 18012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d: No bootstrap required, opened a new log
I20260924 04:39:45.558466 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.544312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.0.98.62:34029)
0924 04:39:45.544932 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.544946 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.544980 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:39:45.545692 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.545710 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.546114 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.546390 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:45.546751 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.547715 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.548931 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.549458 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.549578 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.556813 (+ 7235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:45.556818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:45.556829 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:45.557257 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:45.557312 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:45.557332 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:45.557407 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.557569 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.557819 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
I20260924 04:39:45.559557 18012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b56116dd2574110ac13d32ed4851c2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260924 04:39:45.559775 18012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:45.559854 18012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b56116dd2574110ac13d32ed4851c2d, State: Initialized, Role: FOLLOWER
I20260924 04:39:45.560256 18012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b56116dd2574110ac13d32ed4851c2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260924 04:39:45.560397 18012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:45.560467 18012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:45.560592 18012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:45.561506 18012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b56116dd2574110ac13d32ed4851c2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260924 04:39:45.561839 18012 leader_election.cc:302] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b56116dd2574110ac13d32ed4851c2d; no voters:
I20260924 04:39:45.562119 18012 leader_election.cc:288] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:45.562263 18017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:45.562479 18017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [term 1 LEADER]: Becoming Leader. State: Replica: 0b56116dd2574110ac13d32ed4851c2d, State: Running, Role: LEADER
I20260924 04:39:45.562824 18017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b56116dd2574110ac13d32ed4851c2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260924 04:39:45.563257 18012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:45.564283 18018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b56116dd2574110ac13d32ed4851c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b56116dd2574110ac13d32ed4851c2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260924 04:39:45.564419 18018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:45.564778 18026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:45.565819 18019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b56116dd2574110ac13d32ed4851c2d. Latest consensus state: current_term: 1 leader_uuid: "0b56116dd2574110ac13d32ed4851c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b56116dd2574110ac13d32ed4851c2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260924 04:39:45.565920 18019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:45.567775 18026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:45.573196 18026 catalog_manager.cc:1347] Generated new cluster ID: 52b7b461884e4fbcb1f30b95aed6c4d0
I20260924 04:39:45.573253 18026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:45.578361 18026 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:45.579427 18026 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:45.586447 18026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d: Generated new TSK 0
I20260924 04:39:45.587078 18026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:39:45.595914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--builtin_ntp_servers=127.0.98.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:39:45.719242 18036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:45.719574 18036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:45.719653 18036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:45.724682 18036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:45.724751 18036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:45.724830 18036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:39:45.729651 18036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:45.730662 18036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:45.732429 18036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:45.741684 18044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:45.741891 18036 server_base.cc:1034] running on GCE node
W20260924 04:39:45.741649 18041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:45.741657 18042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:45.742435 18036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:45.743104 18036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:45.744287 18036 hybrid_clock.cc:648] HybridClock initialized: now 1790224785744269 us; error 30 us; skew 500 ppm
I20260924 04:39:45.746501 18036 webserver.cc:460] Webserver started at http://127.0.98.1:37441/ using document root <none> and password file <none>
I20260924 04:39:45.747071 18036 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:45.747164 18036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:45.747380 18036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:45.749401 18036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "5da6b3178d684bfebad7bbfec86c6ba0"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "73c5a89e3f383e057a9fd6a4e2dd78bb"
server_key_iv: "a17aa3b04e6103bbc37aec3fc69c60ee"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.749964 18036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5da6b3178d684bfebad7bbfec86c6ba0"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "73c5a89e3f383e057a9fd6a4e2dd78bb"
server_key_iv: "a17aa3b04e6103bbc37aec3fc69c60ee"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.753583 18036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:39:45.755976 18050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.757126 18036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:45.757325 18036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "5da6b3178d684bfebad7bbfec86c6ba0"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "73c5a89e3f383e057a9fd6a4e2dd78bb"
server_key_iv: "a17aa3b04e6103bbc37aec3fc69c60ee"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.757442 18036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:45.776844 18036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:45.777503 18036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:45.777693 18036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:45.778236 18036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:45.779136 18036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:45.779206 18036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.779275 18036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:45.779316 18036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.788036 18036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35009
I20260924 04:39:45.788069 18163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35009 every 8 connection(s)
I20260924 04:39:45.788858 18036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:39:45.793497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 18036
I20260924 04:39:45.793615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:39:45.793923 392 external_mini_cluster.cc:1428] Setting key 59ef82b41512142f50b5fc8ec8f75291
I20260924 04:39:45.795822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--builtin_ntp_servers=127.0.98.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:39:45.798786 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.790605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58031 (local address 127.0.98.62:34029)
0924 04:39:45.790749 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.790752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.791723 (+ 971us) server_negotiation.cc:407] Connection header received
0924 04:39:45.793004 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.793007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.793049 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.793123 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:45.794153 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.794793 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.796070 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.796289 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.796346 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.798288 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:45.798291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:45.798294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:45.798306 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:45.798326 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:45.798342 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:45.798417 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.798581 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.798652 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:39:45.799571 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.790989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:58031)
0924 04:39:45.791585 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.791623 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:45.792811 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:45.793334 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:45.793340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:45.793420 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:45.793977 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.794012 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.794919 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.794922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.795895 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.795911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.796490 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.796493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.796625 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.798036 (+ 1411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:39:45.798037 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:45.798075 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:39:45.798133 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:45.798432 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:45.798442 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:45.798448 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:39:45.798579 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 04:39:45.798835 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260924 04:39:45.800870 18164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34029
I20260924 04:39:45.801095 18164 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:45.801584 18164 heartbeater.cc:511] Master 127.0.98.62:34029 requested a full tablet report, sending...
I20260924 04:39:45.803233 17976 ts_manager.cc:194] Registered new tserver with Master: 5da6b3178d684bfebad7bbfec86c6ba0 (127.0.98.1:35009)
I20260924 04:39:45.804963 17976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58031
W20260924 04:39:45.901473 18167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:45.901712 18167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:45.901748 18167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:45.905774 18167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:45.905825 18167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:45.905876 18167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:39:45.910662 18167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:45.911537 18167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:45.912752 18167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:45.921893 18175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:45.921903 18172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:45.921903 18173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:45.922194 18167 server_base.cc:1034] running on GCE node
I20260924 04:39:45.922627 18167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:45.923213 18167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:45.924396 18167 hybrid_clock.cc:648] HybridClock initialized: now 1790224785924384 us; error 23 us; skew 500 ppm
I20260924 04:39:45.927026 18167 webserver.cc:460] Webserver started at http://127.0.98.2:42261/ using document root <none> and password file <none>
I20260924 04:39:45.927598 18167 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:45.927662 18167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:45.927876 18167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:45.929925 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "d580844ac12e41c9b0beeddb712ac9d4"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "b1efc460b1c05594af28748501136fc9"
server_key_iv: "585e91081f2d9fbfe8d6e1684d48f723"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.930475 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d580844ac12e41c9b0beeddb712ac9d4"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "b1efc460b1c05594af28748501136fc9"
server_key_iv: "585e91081f2d9fbfe8d6e1684d48f723"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.933995 18167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:39:45.936137 18181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.937330 18167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 04:39:45.937444 18167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "d580844ac12e41c9b0beeddb712ac9d4"
format_stamp: "Formatted at 2026-09-24 04:39:45 on dist-test-slave-jlnt"
server_key: "b1efc460b1c05594af28748501136fc9"
server_key_iv: "585e91081f2d9fbfe8d6e1684d48f723"
server_key_version: "encryptionkey@0"
I20260924 04:39:45.937541 18167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:45.956362 18167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:45.956952 18167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:45.957186 18167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:45.957731 18167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:45.958590 18167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:45.958635 18167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.958701 18167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:45.958741 18167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:45.967451 18167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39089
I20260924 04:39:45.967474 18294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39089 every 8 connection(s)
I20260924 04:39:45.968278 18167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:39:45.971318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 18167
I20260924 04:39:45.971436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:39:45.971699 392 external_mini_cluster.cc:1428] Setting key 9bc5ee4a9bea7fbe85025eaf2b3945e3
I20260924 04:39:45.973655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--builtin_ntp_servers=127.0.98.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:39:45.978704 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.970139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41529 (local address 127.0.98.62:34029)
0924 04:39:45.970272 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:45.970276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:45.971237 (+ 961us) server_negotiation.cc:407] Connection header received
0924 04:39:45.972759 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:45.972761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:45.972801 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:45.972900 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:45.973871 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.974381 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.976080 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.976312 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.976425 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.978218 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:45.978222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:45.978224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:45.978237 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:45.978258 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:45.978274 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:45.978348 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:45.978510 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 04:39:45.978571 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:39:45.979494 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:45.970497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:41529)
0924 04:39:45.971096 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:45.971133 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:45.972555 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:45.973085 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:45.973096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:45.973223 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:45.973679 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.973691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.974509 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.974511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.975942 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:45.975952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:45.976461 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:45.976465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:45.976641 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:45.977958 (+ 1317us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:39:45.977966 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:45.978004 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:39:45.978064 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:45.978362 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:45.978371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:45.978376 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:39:45.978510 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 04:39:45.978768 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
I20260924 04:39:45.980564 18295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34029
I20260924 04:39:45.980799 18295 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:45.981345 18295 heartbeater.cc:511] Master 127.0.98.62:34029 requested a full tablet report, sending...
I20260924 04:39:45.982363 17976 ts_manager.cc:194] Registered new tserver with Master: d580844ac12e41c9b0beeddb712ac9d4 (127.0.98.2:39089)
I20260924 04:39:45.983215 17976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41529
W20260924 04:39:46.081220 18298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:46.081449 18298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:46.081485 18298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:46.085338 18298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:46.085387 18298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:46.085438 18298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:39:46.090058 18298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:46.090924 18298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:46.092114 18298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:46.101001 18304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:46.101008 18303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:46.101011 18306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:46.101099 18298 server_base.cc:1034] running on GCE node
I20260924 04:39:46.101696 18298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:46.102263 18298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:46.103438 18298 hybrid_clock.cc:648] HybridClock initialized: now 1790224786103409 us; error 43 us; skew 500 ppm
I20260924 04:39:46.105718 18298 webserver.cc:460] Webserver started at http://127.0.98.3:36461/ using document root <none> and password file <none>
I20260924 04:39:46.106280 18298 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:46.106370 18298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:46.106592 18298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:46.108496 18298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "f481871cb5bb41a2a18c4c97f7fb75ba"
format_stamp: "Formatted at 2026-09-24 04:39:46 on dist-test-slave-jlnt"
server_key: "6dd980d3ace13277976340c4e60d5008"
server_key_iv: "0eb845dace3971c851f6289aaba267fb"
server_key_version: "encryptionkey@0"
I20260924 04:39:46.109043 18298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f481871cb5bb41a2a18c4c97f7fb75ba"
format_stamp: "Formatted at 2026-09-24 04:39:46 on dist-test-slave-jlnt"
server_key: "6dd980d3ace13277976340c4e60d5008"
server_key_iv: "0eb845dace3971c851f6289aaba267fb"
server_key_version: "encryptionkey@0"
I20260924 04:39:46.112390 18298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:39:46.114653 18312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:46.115712 18298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:39:46.115839 18298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "f481871cb5bb41a2a18c4c97f7fb75ba"
format_stamp: "Formatted at 2026-09-24 04:39:46 on dist-test-slave-jlnt"
server_key: "6dd980d3ace13277976340c4e60d5008"
server_key_iv: "0eb845dace3971c851f6289aaba267fb"
server_key_version: "encryptionkey@0"
I20260924 04:39:46.115959 18298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:46.133097 18298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:46.133831 18298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:46.134016 18298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:46.134531 18298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:46.135371 18298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:46.135438 18298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:46.135504 18298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:46.135553 18298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:46.144162 18298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39379
I20260924 04:39:46.144187 18425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39379 every 8 connection(s)
I20260924 04:39:46.144963 18298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:39:46.149376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 18298
I20260924 04:39:46.149488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:39:46.149718 392 external_mini_cluster.cc:1428] Setting key 47f3aaf986cb185dbd496aeecc277a22
I20260924 04:39:46.154484 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.146660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33415 (local address 127.0.98.62:34029)
0924 04:39:46.146815 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:46.146820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:46.147755 (+ 935us) server_negotiation.cc:407] Connection header received
0924 04:39:46.149011 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:46.149014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:46.149057 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:46.149130 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:46.150129 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.150651 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.151863 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.152092 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.152202 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.154007 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:46.154010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:46.154013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:46.154025 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:46.154044 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:46.154059 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:46.154137 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:46.154296 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 04:39:46.154350 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:39:46.155217 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.147022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:33415)
0924 04:39:46.147624 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:46.147661 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:46.148832 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:46.149346 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:46.149353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:46.149430 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:46.149992 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.150006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.150780 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.150788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.151744 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.151753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.152235 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.152239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.152395 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.153752 (+ 1357us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:39:46.153753 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:46.153790 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:39:46.153846 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:46.154139 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:46.154151 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:46.154158 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:39:46.154285 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 04:39:46.154548 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260924 04:39:46.156214 18426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34029
I20260924 04:39:46.156396 18426 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:46.156802 18426 heartbeater.cc:511] Master 127.0.98.62:34029 requested a full tablet report, sending...
I20260924 04:39:46.157703 17976 ts_manager.cc:194] Registered new tserver with Master: f481871cb5bb41a2a18c4c97f7fb75ba (127.0.98.3:39379)
I20260924 04:39:46.158237 17976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33415
I20260924 04:39:46.163507 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:39:46.169562 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.165487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.0.98.62:34029)
0924 04:39:46.165616 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:46.165620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:46.165775 (+ 155us) server_negotiation.cc:407] Connection header received
0924 04:39:46.165870 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:46.165872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:46.165905 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:46.165972 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:46.166195 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.166669 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.167414 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.167622 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.167694 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.169197 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:46.169201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:46.169204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:46.169215 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:46.169232 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:46.169247 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:46.169324 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:46.169400 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 04:39:46.169439 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:39:46.173465 17976 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:39:46.176367 17976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:39:46.190443 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.184887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.0.98.1:35009)
0924 04:39:46.185262 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:46.185274 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:46.185300 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:39:46.185377 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:46.185382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:46.185600 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:46.185756 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:46.186097 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.187355 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.189264 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.189766 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.189911 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.190102 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:46.190194 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 04:39:46.190288 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":153,"threads_started":1}
I20260924 04:39:46.190506 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.184221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35009 (local address 127.0.0.1:49544)
0924 04:39:46.184574 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:46.184623 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:46.184745 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:46.185821 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:46.185826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:46.185850 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:46.185977 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.185984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.187481 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.187486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.189122 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.189130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.189907 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.189911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.190015 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.190050 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:39:46.190155 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:39:46.190269 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":37,"thread_start_us":79,"threads_started":1}
I20260924 04:39:46.190505 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.184347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39089 (local address 127.0.0.1:34164)
0924 04:39:46.185309 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:46.185324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:46.185406 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:46.186292 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:46.186298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:46.186321 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:46.186423 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.186428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.187784 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.187788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.188996 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.189005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.190045 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.190048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.190150 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.190173 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:39:46.190237 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:39:46.190303 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":47,"thread_start_us":90,"threads_started":1}
I20260924 04:39:46.190732 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.184632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39379 (local address 127.0.0.1:58116)
0924 04:39:46.185582 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:46.185595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:46.185670 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:46.186379 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:46.186382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:46.186394 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:46.186489 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.186497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.187597 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.187602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.188814 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.188823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.189928 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.189933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.190376 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.190399 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:39:46.190472 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:39:46.190549 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":298,"thread_start_us":92,"threads_started":1}
I20260924 04:39:46.191239 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.185052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.0.98.3:39379)
0924 04:39:46.185934 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:46.185947 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:46.185969 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:39:46.186042 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:46.186049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:46.186217 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:46.186621 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:46.186725 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.187481 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.188937 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.189770 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.190112 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.190297 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:46.190984 (+ 687us) server_negotiation.cc:299] Negotiation successful
0924 04:39:46.191092 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":133,"threads_started":1}
I20260924 04:39:46.191367 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.185434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.0.98.2:39089)
0924 04:39:46.185852 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:46.185866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:46.185890 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:39:46.185959 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:46.185968 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:46.186145 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:46.186297 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:46.186538 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.187665 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.189116 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.189928 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.190878 (+ 950us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.191070 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:46.191122 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 04:39:46.191227 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":132,"threads_started":1}
I20260924 04:39:46.193789 18229 tablet_service.cc:1467] Processing CreateTablet for tablet ed5f9fa93d0349d5967b95b5f80dcb9f (DEFAULT_TABLE table=test-table [id=cb300f5029b64511ba0c4e5d575f1981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:46.193892 18360 tablet_service.cc:1467] Processing CreateTablet for tablet ed5f9fa93d0349d5967b95b5f80dcb9f (DEFAULT_TABLE table=test-table [id=cb300f5029b64511ba0c4e5d575f1981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:46.193892 18098 tablet_service.cc:1467] Processing CreateTablet for tablet ed5f9fa93d0349d5967b95b5f80dcb9f (DEFAULT_TABLE table=test-table [id=cb300f5029b64511ba0c4e5d575f1981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:46.194767 18229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5f9fa93d0349d5967b95b5f80dcb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:46.194885 18098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5f9fa93d0349d5967b95b5f80dcb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:46.194885 18360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5f9fa93d0349d5967b95b5f80dcb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:46.200577 18443 tablet_bootstrap.cc:492] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: Bootstrap starting.
I20260924 04:39:46.202107 18445 tablet_bootstrap.cc:492] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0: Bootstrap starting.
I20260924 04:39:46.202106 18444 tablet_bootstrap.cc:492] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: Bootstrap starting.
I20260924 04:39:46.202782 18443 tablet_bootstrap.cc:654] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:46.203490 18443 log.cc:824] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:46.204200 18445 tablet_bootstrap.cc:654] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:46.204200 18444 tablet_bootstrap.cc:654] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:46.205039 18444 log.cc:824] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:46.205039 18445 log.cc:824] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:46.205651 18443 tablet_bootstrap.cc:492] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: No bootstrap required, opened a new log
I20260924 04:39:46.205884 18443 ts_tablet_manager.cc:1397] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:39:46.206962 18444 tablet_bootstrap.cc:492] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: No bootstrap required, opened a new log
I20260924 04:39:46.206971 18445 tablet_bootstrap.cc:492] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0: No bootstrap required, opened a new log
I20260924 04:39:46.207140 18444 ts_tablet_manager.cc:1397] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:39:46.207140 18445 ts_tablet_manager.cc:1397] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 04:39:46.209067 18443 raft_consensus.cc:348] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } }
I20260924 04:39:46.209375 18443 raft_consensus.cc:374] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:46.209460 18443 raft_consensus.cc:729] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d580844ac12e41c9b0beeddb712ac9d4, State: Initialized, Role: FOLLOWER
I20260924 04:39:46.209903 18443 consensus_queue.cc:260] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } }
I20260924 04:39:46.210507 18295 heartbeater.cc:503] Master 127.0.98.62:34029 was elected leader, sending a full tablet report...
I20260924 04:39:46.210842 18443 ts_tablet_manager.cc:1428] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:39:46.211376 18445 raft_consensus.cc:348] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } }
I20260924 04:39:46.211376 18444 raft_consensus.cc:348] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } }
I20260924 04:39:46.211621 18444 raft_consensus.cc:374] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:46.211694 18445 raft_consensus.cc:374] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:46.211709 18444 raft_consensus.cc:729] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f481871cb5bb41a2a18c4c97f7fb75ba, State: Initialized, Role: FOLLOWER
I20260924 04:39:46.211758 18445 raft_consensus.cc:729] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da6b3178d684bfebad7bbfec86c6ba0, State: Initialized, Role: FOLLOWER
I20260924 04:39:46.212137 18444 consensus_queue.cc:260] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } }
I20260924 04:39:46.212473 18445 consensus_queue.cc:260] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } }
I20260924 04:39:46.213130 18164 heartbeater.cc:503] Master 127.0.98.62:34029 was elected leader, sending a full tablet report...
I20260924 04:39:46.213363 18445 ts_tablet_manager.cc:1428] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 04:39:46.213701 18426 heartbeater.cc:503] Master 127.0.98.62:34029 was elected leader, sending a full tablet report...
I20260924 04:39:46.214720 18444 ts_tablet_manager.cc:1428] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260924 04:39:46.219141 18296 tablet.cc:2369] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:39:46.264384 18449 raft_consensus.cc:482] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:46.264597 18449 raft_consensus.cc:504] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } }
I20260924 04:39:46.265909 18449 leader_election.cc:288] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5da6b3178d684bfebad7bbfec86c6ba0 (127.0.98.1:35009), f481871cb5bb41a2a18c4c97f7fb75ba (127.0.98.3:39379)
I20260924 04:39:46.270768 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.266206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47719 (local address 127.0.98.1:35009)
0924 04:39:46.266323 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:46.266327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:46.266353 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:39:46.266514 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:46.266517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:46.266572 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:46.266655 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:46.266984 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.267755 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.269423 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.270047 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.270157 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.270217 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:46.270570 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 04:39:46.270640 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:39:46.270768 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.266255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42209 (local address 127.0.98.3:39379)
0924 04:39:46.266397 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:46.266401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:46.266748 (+ 347us) server_negotiation.cc:407] Connection header received
0924 04:39:46.266842 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:46.266849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:46.266885 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:46.266973 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:46.267220 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.267923 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.269423 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.270057 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.270157 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.270217 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:46.270570 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 04:39:46.270640 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:39:46.270810 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.266101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35009 (local address 127.0.98.2:47719)
0924 04:39:46.266264 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:46.266276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:46.266387 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:46.266691 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:46.266696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:46.266720 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:46.266873 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.266878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.267872 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.267874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.269297 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.269305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.270174 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.270178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.270406 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.270458 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 04:39:46.270538 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:39:46.270654 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":162,"spinlock_wait_cycles":9984}
I20260924 04:39:46.271004 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.266114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39379 (local address 127.0.98.2:42209)
0924 04:39:46.266648 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:46.266664 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:46.266765 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:46.267006 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:46.267008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:46.267025 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:46.267122 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.267126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.268008 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.268010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.269293 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:46.269305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.270177 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.270180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:46.270370 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.270458 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 04:39:46.270538 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:39:46.270654 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":391,"spinlock_wait_cycles":2304,"thread_start_us":142,"threads_started":1}
I20260924 04:39:46.271435 18118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5f9fa93d0349d5967b95b5f80dcb9f" candidate_uuid: "d580844ac12e41c9b0beeddb712ac9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" is_pre_election: true
I20260924 04:39:46.271729 18118 raft_consensus.cc:2454] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d580844ac12e41c9b0beeddb712ac9d4 in term 0.
I20260924 04:39:46.271708 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5f9fa93d0349d5967b95b5f80dcb9f" candidate_uuid: "d580844ac12e41c9b0beeddb712ac9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" is_pre_election: true
I20260924 04:39:46.271979 18380 raft_consensus.cc:2454] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d580844ac12e41c9b0beeddb712ac9d4 in term 0.
I20260924 04:39:46.272245 18183 leader_election.cc:302] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da6b3178d684bfebad7bbfec86c6ba0, d580844ac12e41c9b0beeddb712ac9d4; no voters:
I20260924 04:39:46.272490 18449 raft_consensus.cc:2781] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:39:46.272560 18449 raft_consensus.cc:482] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:39:46.272589 18449 raft_consensus.cc:3037] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:46.273679 18449 raft_consensus.cc:504] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } }
I20260924 04:39:46.274024 18449 leader_election.cc:288] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [CANDIDATE]: Term 1 election: Requested vote from peers 5da6b3178d684bfebad7bbfec86c6ba0 (127.0.98.1:35009), f481871cb5bb41a2a18c4c97f7fb75ba (127.0.98.3:39379)
I20260924 04:39:46.274302 18118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5f9fa93d0349d5967b95b5f80dcb9f" candidate_uuid: "d580844ac12e41c9b0beeddb712ac9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da6b3178d684bfebad7bbfec86c6ba0"
I20260924 04:39:46.274307 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5f9fa93d0349d5967b95b5f80dcb9f" candidate_uuid: "d580844ac12e41c9b0beeddb712ac9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba"
I20260924 04:39:46.274431 18118 raft_consensus.cc:3037] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:46.274441 18380 raft_consensus.cc:3037] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:46.275411 18380 raft_consensus.cc:2454] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d580844ac12e41c9b0beeddb712ac9d4 in term 1.
I20260924 04:39:46.275414 18118 raft_consensus.cc:2454] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d580844ac12e41c9b0beeddb712ac9d4 in term 1.
I20260924 04:39:46.275696 18185 leader_election.cc:302] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d580844ac12e41c9b0beeddb712ac9d4, f481871cb5bb41a2a18c4c97f7fb75ba; no voters:
I20260924 04:39:46.275876 18449 raft_consensus.cc:2781] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:46.276079 18449 raft_consensus.cc:686] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 1 LEADER]: Becoming Leader. State: Replica: d580844ac12e41c9b0beeddb712ac9d4, State: Running, Role: LEADER
I20260924 04:39:46.276381 18449 consensus_queue.cc:237] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } }
I20260924 04:39:46.279317 17974 catalog_manager.cc:5495] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d580844ac12e41c9b0beeddb712ac9d4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d580844ac12e41c9b0beeddb712ac9d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d580844ac12e41c9b0beeddb712ac9d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } health_report { overall_health: UNKNOWN } } }
W20260924 04:39:46.289938 18165 tablet.cc:2369] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:39:46.308801 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.305385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.0.98.2:39089)
0924 04:39:46.305500 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:46.305504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:46.305544 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:39:46.305727 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:46.305729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:46.305780 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:46.305915 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:39:46.306142 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.306624 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.307462 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:46.307678 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:46.307787 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:46.308071 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:39:46.308431 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:39:46.308557 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:46.308618 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 04:39:46.308666 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:39:46.314030 18380 raft_consensus.cc:1261] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [term 1 FOLLOWER]: Refusing update from remote peer d580844ac12e41c9b0beeddb712ac9d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:46.314030 18118 raft_consensus.cc:1261] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [term 1 FOLLOWER]: Refusing update from remote peer d580844ac12e41c9b0beeddb712ac9d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:46.314666 18449 consensus_queue.cc:1035] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f481871cb5bb41a2a18c4c97f7fb75ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:46.314860 18453 consensus_queue.cc:1035] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da6b3178d684bfebad7bbfec86c6ba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:46.329651 17976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:46.329836 17976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:46.332084 17976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b56116dd2574110ac13d32ed4851c2d: Sending DeleteTablet for 3 replicas of tablet ed5f9fa93d0349d5967b95b5f80dcb9f
I20260924 04:39:46.332767 18098 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5f9fa93d0349d5967b95b5f80dcb9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:46 UTC) from {username='slave'} at 127.0.0.1:49544
I20260924 04:39:46.332830 18229 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5f9fa93d0349d5967b95b5f80dcb9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:46 UTC) from {username='slave'} at 127.0.0.1:34164
I20260924 04:39:46.332949 18360 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5f9fa93d0349d5967b95b5f80dcb9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:46 UTC) from {username='slave'} at 127.0.0.1:58116
I20260924 04:39:46.333312 18466 tablet_replica.cc:330] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0: stopping tablet replica
I20260924 04:39:46.333335 18468 tablet_replica.cc:330] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: stopping tablet replica
I20260924 04:39:46.333627 18468 raft_consensus.cc:2229] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:46.333627 18466 raft_consensus.cc:2229] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:46.333911 18466 raft_consensus.cc:2258] T ed5f9fa93d0349d5967b95b5f80dcb9f P 5da6b3178d684bfebad7bbfec86c6ba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:46.333911 18468 raft_consensus.cc:2258] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:46.334307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 18036
I20260924 04:39:46.334910 18467 tablet_replica.cc:330] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: stopping tablet replica
I20260924 04:39:46.335196 18467 raft_consensus.cc:2229] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:39:46.335486 18467 raft_consensus.cc:2258] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:46.336927 18468 ts_tablet_manager.cc:1905] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:46.337610 18467 ts_tablet_manager.cc:1905] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:46.339854 18468 ts_tablet_manager.cc:1918] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:46.340018 18468 log.cc:1196] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/ed5f9fa93d0349d5967b95b5f80dcb9f
I20260924 04:39:46.340296 18467 ts_tablet_manager.cc:1918] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:46.340396 18467 log.cc:1196] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/ed5f9fa93d0349d5967b95b5f80dcb9f
I20260924 04:39:46.340524 18468 ts_tablet_manager.cc:1939] T ed5f9fa93d0349d5967b95b5f80dcb9f P f481871cb5bb41a2a18c4c97f7fb75ba: Deleting consensus metadata
I20260924 04:39:46.340701 18467 ts_tablet_manager.cc:1939] T ed5f9fa93d0349d5967b95b5f80dcb9f P d580844ac12e41c9b0beeddb712ac9d4: Deleting consensus metadata
I20260924 04:39:46.341778 17963 catalog_manager.cc:4841] TS f481871cb5bb41a2a18c4c97f7fb75ba (127.0.98.3:39379): tablet ed5f9fa93d0349d5967b95b5f80dcb9f (table test-table [id=cb300f5029b64511ba0c4e5d575f1981]) successfully deleted
I20260924 04:39:46.342304 17962 catalog_manager.cc:4841] TS d580844ac12e41c9b0beeddb712ac9d4 (127.0.98.2:39089): tablet ed5f9fa93d0349d5967b95b5f80dcb9f (table test-table [id=cb300f5029b64511ba0c4e5d575f1981]) successfully deleted
W20260924 04:39:46.343817 17961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35009 (error 108)
I20260924 04:39:46.344552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 18167
I20260924 04:39:46.344847 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:46.344557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35009 (local address 127.0.0.1:49548)
0924 04:39:46.344685 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:46.344755 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 04:39:46.345207 17961 catalog_manager.cc:4584] TS 5da6b3178d684bfebad7bbfec86c6ba0 (127.0.98.1:35009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed5f9fa93d0349d5967b95b5f80dcb9f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35009: connect: Connection refused (error 111)
I20260924 04:39:46.349843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 18298
I20260924 04:39:46.356453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 17945
2026-09-24T04:39:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1007 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:39:46 dist-test-slave-jlnt krb5kdc[18472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:39:46 dist-test-slave-jlnt krb5kdc[18472](info): set up 2 sockets
Sep 24 04:39:46 dist-test-slave-jlnt krb5kdc[18472](info): commencing operation
krb5kdc: starting...
W20260924 04:39:47.384868 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:39:47 dist-test-slave-jlnt krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:39:47.504475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43021
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:47.612140 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:47.612396 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:47.612435 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:47.616446 18485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:47.616508 18485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:47.616526 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:47.616542 18485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:47.616557 18485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:47.621733 18485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:47.622686 18485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:47.623690 18485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:47.632320 18491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:47.632275 18493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:47.632269 18490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:47.632599 18485 server_base.cc:1034] running on GCE node
I20260924 04:39:47.633214 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:47.634229 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:47.635408 18485 hybrid_clock.cc:648] HybridClock initialized: now 1790224787635378 us; error 41 us; skew 500 ppm
Sep 24 04:39:47 dist-test-slave-jlnt krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:47.638734 18485 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:39:47.639652 18485 webserver.cc:460] Webserver started at http://127.0.98.62:44233/ using document root <none> and password file <none>
I20260924 04:39:47.640215 18485 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:47.640280 18485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:47.640457 18485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:47.642584 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "072bcc849d1c4f1ab21bcd9b442225bf"
format_stamp: "Formatted at 2026-09-24 04:39:47 on dist-test-slave-jlnt"
server_key: "9981d62ae1b47644d6a5de1a20d8268a"
server_key_iv: "026c289f5d561ad0222a1f52f15146f9"
server_key_version: "encryptionkey@0"
I20260924 04:39:47.643134 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "072bcc849d1c4f1ab21bcd9b442225bf"
format_stamp: "Formatted at 2026-09-24 04:39:47 on dist-test-slave-jlnt"
server_key: "9981d62ae1b47644d6a5de1a20d8268a"
server_key_iv: "026c289f5d561ad0222a1f52f15146f9"
server_key_version: "encryptionkey@0"
I20260924 04:39:47.647064 18485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:39:47.649514 18500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:47.650743 18485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:47.650842 18485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "072bcc849d1c4f1ab21bcd9b442225bf"
format_stamp: "Formatted at 2026-09-24 04:39:47 on dist-test-slave-jlnt"
server_key: "9981d62ae1b47644d6a5de1a20d8268a"
server_key_iv: "026c289f5d561ad0222a1f52f15146f9"
server_key_version: "encryptionkey@0"
I20260924 04:39:47.650923 18485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:47.662947 18485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:47.668108 18485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:47.668251 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:47.674813 18485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43021
I20260924 04:39:47.675077 18552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43021 every 8 connection(s)
I20260924 04:39:47.675733 18485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:47.678422 18553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:47.679787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 18485
I20260924 04:39:47.679912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:47.680174 392 external_mini_cluster.cc:1428] Setting key b3abfc00cb9e5c6efc8ff4300af20ca0
I20260924 04:39:47.684737 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf: Bootstrap starting.
I20260924 04:39:47.687172 18553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:47.687873 18553 log.cc:824] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:47.689921 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf: No bootstrap required, opened a new log
Sep 24 04:39:47 dist-test-slave-jlnt krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:47.694307 18553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072bcc849d1c4f1ab21bcd9b442225bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } }
I20260924 04:39:47.694566 18553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:47.694625 18553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072bcc849d1c4f1ab21bcd9b442225bf, State: Initialized, Role: FOLLOWER
I20260924 04:39:47.695096 18553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072bcc849d1c4f1ab21bcd9b442225bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } }
I20260924 04:39:47.695253 18553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:47.695319 18553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:47.695407 18553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:47.696422 18553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072bcc849d1c4f1ab21bcd9b442225bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } }
I20260924 04:39:47.696756 18553 leader_election.cc:302] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 072bcc849d1c4f1ab21bcd9b442225bf; no voters:
I20260924 04:39:47.697043 18553 leader_election.cc:288] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:47.697190 18558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:47.697227 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:47.681461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.0.98.62:43021)
0924 04:39:47.681875 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:47.681892 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:47.681932 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:39:47.682704 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:47.682727 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:47.683147 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:47.683487 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:47.684209 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:47.685614 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:47.686419 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:47.687087 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:47.687250 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:47.690663 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:47.690669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:47.690685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:47.690712 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:47.694248 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:47.694686 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:47.694692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:47.694699 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:47.694827 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:47.695109 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:47.695115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:47.695117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:47.695608 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:47.695795 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:47.696014 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 04:39:47.696303 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
I20260924 04:39:47.697438 18558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [term 1 LEADER]: Becoming Leader. State: Replica: 072bcc849d1c4f1ab21bcd9b442225bf, State: Running, Role: LEADER
I20260924 04:39:47.697717 18558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072bcc849d1c4f1ab21bcd9b442225bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } }
I20260924 04:39:47.698807 18558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 072bcc849d1c4f1ab21bcd9b442225bf. Latest consensus state: current_term: 1 leader_uuid: "072bcc849d1c4f1ab21bcd9b442225bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072bcc849d1c4f1ab21bcd9b442225bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } } }
I20260924 04:39:47.698926 18558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:47.698982 18553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:47.699517 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "072bcc849d1c4f1ab21bcd9b442225bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072bcc849d1c4f1ab21bcd9b442225bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } } }
I20260924 04:39:47.699613 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:47.700552 18562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:47.703234 18562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:47.708282 18562 catalog_manager.cc:1347] Generated new cluster ID: 87373fce83a445f49ad2681619cbc078
I20260924 04:39:47.708341 18562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:47.747370 18562 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:47.748636 18562 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:47.756181 18562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf: Generated new TSK 0
I20260924 04:39:47.756937 18562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:39:47.789091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:47.894675 18579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:47.894896 18579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:47.894935 18579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:47.898815 18579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:47.898865 18579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:47.898921 18579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:39:47.904167 18579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:47.905036 18579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:47.905961 18579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:47.914875 18587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:47.914901 18584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:47.914943 18585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:47.915025 18579 server_base.cc:1034] running on GCE node
I20260924 04:39:47.915463 18579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:47.915997 18579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:47.917199 18579 hybrid_clock.cc:648] HybridClock initialized: now 1790224787917160 us; error 42 us; skew 500 ppm
Sep 24 04:39:47 dist-test-slave-jlnt krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:47.920743 18579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:39:47.921615 18579 webserver.cc:460] Webserver started at http://127.0.98.1:32833/ using document root <none> and password file <none>
I20260924 04:39:47.922217 18579 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:47.922274 18579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:47.922479 18579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:47.924379 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "dcda3f8ee7314651bb9e25c602f807a5"
format_stamp: "Formatted at 2026-09-24 04:39:47 on dist-test-slave-jlnt"
server_key: "a885a2083e54830a4ad72020168c7ebb"
server_key_iv: "7ef0972740c0818c9668960f05c6c9a2"
server_key_version: "encryptionkey@0"
I20260924 04:39:47.924914 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dcda3f8ee7314651bb9e25c602f807a5"
format_stamp: "Formatted at 2026-09-24 04:39:47 on dist-test-slave-jlnt"
server_key: "a885a2083e54830a4ad72020168c7ebb"
server_key_iv: "7ef0972740c0818c9668960f05c6c9a2"
server_key_version: "encryptionkey@0"
I20260924 04:39:47.928460 18579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:39:47.930827 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:47.932065 18579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 04:39:47.932181 18579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "dcda3f8ee7314651bb9e25c602f807a5"
format_stamp: "Formatted at 2026-09-24 04:39:47 on dist-test-slave-jlnt"
server_key: "a885a2083e54830a4ad72020168c7ebb"
server_key_iv: "7ef0972740c0818c9668960f05c6c9a2"
server_key_version: "encryptionkey@0"
I20260924 04:39:47.932281 18579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:47.951579 18579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:47.956404 18579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:47.956590 18579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:47.957129 18579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:47.957904 18579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:47.957948 18579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:47.958012 18579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:47.958042 18579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:47.966373 18579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44981
I20260924 04:39:47.966413 18707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44981 every 8 connection(s)
I20260924 04:39:47.967239 18579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:39:47.974752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 18579
I20260924 04:39:47.974895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:39:47.975235 392 external_mini_cluster.cc:1428] Setting key 82af8822147ea92060fd0a0a3ca65491
Sep 24 04:39:47 dist-test-slave-jlnt krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:47.983639 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:47.968865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35907 (local address 127.0.98.62:43021)
0924 04:39:47.969002 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:47.969005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:47.969879 (+ 874us) server_negotiation.cc:407] Connection header received
0924 04:39:47.971276 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:47.971279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:47.971324 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:47.971408 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:47.972500 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:47.973026 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:47.974302 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:47.974533 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:47.974614 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:47.978239 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:47.978247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:47.978254 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:47.978299 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:47.981330 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:47.981881 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:47.981886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:47.981890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:47.981990 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:47.982339 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:47.982345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:47.982351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:47.982638 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:47.982765 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:47.983250 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 04:39:47.983419 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:39:47.984156 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:47.969230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43021 (local address 127.0.98.1:35907)
0924 04:39:47.969671 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:47.969719 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:47.971079 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:47.971543 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:47.971550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:47.971890 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:47.972359 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:47.972373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:47.973192 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:47.973196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:47.974190 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:47.974200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:47.974698 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:47.974703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:47.974994 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:47.976333 (+ 1339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:47.976350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:47.978068 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:47.981478 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:47.981483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:47.981493 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:47.981749 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:47.982091 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:47.982093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:47.982095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:47.982215 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:47.982747 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:47.982752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:47.982995 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 04:39:47.983132 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 04:39:47.983357 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260924 04:39:47.985395 18708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43021
I20260924 04:39:47.985668 18708 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:39:47.986265 18708 heartbeater.cc:511] Master 127.0.98.62:43021 requested a full tablet report, sending...
I20260924 04:39:47.987864 18517 ts_manager.cc:194] Registered new tserver with Master: dcda3f8ee7314651bb9e25c602f807a5 (127.0.98.1:44981)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:39:47.989224 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:39:48.004536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:48.117328 18713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:48.117599 18713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:48.117673 18713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:48.121598 18713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:48.121686 18713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:48.121780 18713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:39:48.126963 18713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:48.127982 18713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:48.128993 18713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:48.138906 18721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:48.138959 18718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:48.138980 18713 server_base.cc:1034] running on GCE node
W20260924 04:39:48.138912 18719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:48.139521 18713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:48.140143 18713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:48.141345 18713 hybrid_clock.cc:648] HybridClock initialized: now 1790224788141314 us; error 33 us; skew 500 ppm
Sep 24 04:39:48 dist-test-slave-jlnt krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:48.144650 18713 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:39:48.145689 18713 webserver.cc:460] Webserver started at http://127.0.98.2:43739/ using document root <none> and password file <none>
I20260924 04:39:48.146377 18713 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:48.146478 18713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:48.146723 18713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:48.148838 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "d07167221c0f4065aa5a3ecd02cd040d"
format_stamp: "Formatted at 2026-09-24 04:39:48 on dist-test-slave-jlnt"
server_key: "b2a3d2993a468fdcbc9566e2ba840f4a"
server_key_iv: "52b6cd1787b02e2953644aec8061fc27"
server_key_version: "encryptionkey@0"
I20260924 04:39:48.149470 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d07167221c0f4065aa5a3ecd02cd040d"
format_stamp: "Formatted at 2026-09-24 04:39:48 on dist-test-slave-jlnt"
server_key: "b2a3d2993a468fdcbc9566e2ba840f4a"
server_key_iv: "52b6cd1787b02e2953644aec8061fc27"
server_key_version: "encryptionkey@0"
I20260924 04:39:48.153311 18713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:39:48.155685 18728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:48.156798 18713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:48.156955 18713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "d07167221c0f4065aa5a3ecd02cd040d"
format_stamp: "Formatted at 2026-09-24 04:39:48 on dist-test-slave-jlnt"
server_key: "b2a3d2993a468fdcbc9566e2ba840f4a"
server_key_iv: "52b6cd1787b02e2953644aec8061fc27"
server_key_version: "encryptionkey@0"
I20260924 04:39:48.157060 18713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:48.202018 18713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:48.205058 18713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:48.205307 18713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:48.205888 18713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:48.206713 18713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:48.206786 18713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:48.206853 18713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:48.206887 18713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:48.215572 18713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34313
I20260924 04:39:48.215600 18841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34313 every 8 connection(s)
I20260924 04:39:48.216430 18713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:39:48.220911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 18713
I20260924 04:39:48.221041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:39:48.221350 392 external_mini_cluster.cc:1428] Setting key 9889f8b3106ca5f696bf4cc890ae2560
Sep 24 04:39:48 dist-test-slave-jlnt krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:39:48.231428 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.218232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59329 (local address 127.0.98.62:43021)
0924 04:39:48.218376 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:48.218379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:48.219190 (+ 811us) server_negotiation.cc:407] Connection header received
0924 04:39:48.220544 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:48.220547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:48.220601 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:48.220708 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:48.222084 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.222835 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.224577 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.224980 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.225101 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.227804 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:48.227808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:48.227810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:48.227830 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:48.229373 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:48.229888 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:48.229894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:48.229895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:48.229978 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:48.230306 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:48.230311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:48.230312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:48.230508 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:48.230614 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:48.231142 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 04:39:48.231261 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:39:48.232280 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.218636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43021 (local address 127.0.98.2:59329)
0924 04:39:48.219050 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:48.219089 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:48.220345 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:48.220882 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:48.220889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:48.221331 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:48.221906 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.221925 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.222982 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.222985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.224453 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.224467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.225155 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.225189 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.225361 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.225984 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:48.226008 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:48.227639 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:48.229497 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:48.229502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:48.229516 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:48.229773 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:48.230071 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:48.230075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:48.230077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:48.230210 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:48.230613 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:48.230619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:48.230891 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 04:39:48.231102 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 04:39:48.231397 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:39:48.233425 18842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43021
I20260924 04:39:48.233670 18842 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:48.234114 18842 heartbeater.cc:511] Master 127.0.98.62:43021 requested a full tablet report, sending...
I20260924 04:39:48.235136 18517 ts_manager.cc:194] Registered new tserver with Master: d07167221c0f4065aa5a3ecd02cd040d (127.0.98.2:34313)
I20260924 04:39:48.235973 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:39:48.248682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:48.354269 18847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:48.354496 18847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:48.354568 18847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:48.358985 18847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:48.359059 18847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:48.359136 18847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:39:48.365602 18847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:48.366560 18847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:48.367578 18847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:48.376684 18855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:48.376734 18852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:48.376685 18853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:48.377053 18847 server_base.cc:1034] running on GCE node
I20260924 04:39:48.377485 18847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:48.378063 18847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:48.379240 18847 hybrid_clock.cc:648] HybridClock initialized: now 1790224788379225 us; error 26 us; skew 500 ppm
Sep 24 04:39:48 dist-test-slave-jlnt krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:48.381982 18847 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:39:48.382705 18847 webserver.cc:460] Webserver started at http://127.0.98.3:33235/ using document root <none> and password file <none>
I20260924 04:39:48.383186 18847 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:48.383239 18847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:48.383386 18847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:48.385282 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "2ad3a4f015a54912885fa7587955aa71"
format_stamp: "Formatted at 2026-09-24 04:39:48 on dist-test-slave-jlnt"
server_key: "7a9d833e6eee3a0391ba5587ef8569f7"
server_key_iv: "9ae951001c9ebc7f61d7ba05fde62949"
server_key_version: "encryptionkey@0"
I20260924 04:39:48.385773 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2ad3a4f015a54912885fa7587955aa71"
format_stamp: "Formatted at 2026-09-24 04:39:48 on dist-test-slave-jlnt"
server_key: "7a9d833e6eee3a0391ba5587ef8569f7"
server_key_iv: "9ae951001c9ebc7f61d7ba05fde62949"
server_key_version: "encryptionkey@0"
I20260924 04:39:48.389340 18847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:39:48.391517 18862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:48.392663 18847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:48.392750 18847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "2ad3a4f015a54912885fa7587955aa71"
format_stamp: "Formatted at 2026-09-24 04:39:48 on dist-test-slave-jlnt"
server_key: "7a9d833e6eee3a0391ba5587ef8569f7"
server_key_iv: "9ae951001c9ebc7f61d7ba05fde62949"
server_key_version: "encryptionkey@0"
I20260924 04:39:48.392824 18847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:48.409680 18847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:48.412712 18847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:48.412851 18847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:48.413395 18847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:48.414109 18847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:48.414160 18847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:48.414196 18847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:48.414216 18847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:48.422634 18847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37067
I20260924 04:39:48.422657 18975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37067 every 8 connection(s)
I20260924 04:39:48.423450 18847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:39:48.424019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 18847
I20260924 04:39:48.424165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:39:48.424476 392 external_mini_cluster.cc:1428] Setting key 50b7a91444c41029bb907fadc5af43dd
Sep 24 04:39:48 dist-test-slave-jlnt krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:48.436213 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.425336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53957 (local address 127.0.98.62:43021)
0924 04:39:48.425476 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:48.425483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:48.426452 (+ 969us) server_negotiation.cc:407] Connection header received
0924 04:39:48.427727 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:48.427730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:48.427772 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:48.427840 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:48.428875 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.429473 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.430641 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.430838 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.430910 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.433062 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:48.433065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:48.433067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:48.433087 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:48.434536 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:48.434948 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:48.434950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:48.434952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:48.435019 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:48.435337 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:48.435340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:48.435341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:48.435549 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:48.435624 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:48.436017 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 04:39:48.436100 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:39:48.437076 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.425788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43021 (local address 127.0.98.3:53957)
0924 04:39:48.426273 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:48.426325 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:48.427535 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:48.427975 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:48.427981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:48.428304 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:48.428747 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.428759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.429581 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.429584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.430543 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.430552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.430952 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.430955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.431091 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.431570 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:48.431585 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:48.432913 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:48.434625 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:48.434629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:48.434638 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:48.434862 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:48.435093 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:48.435095 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:48.435096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:48.435214 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:48.435621 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:48.435625 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:48.435826 (+ 201us) client_negotiation.cc:769] Sending connection context
0924 04:39:48.435980 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 04:39:48.436204 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260924 04:39:48.438265 18976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43021
I20260924 04:39:48.438490 18976 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:48.438884 18976 heartbeater.cc:511] Master 127.0.98.62:43021 requested a full tablet report, sending...
I20260924 04:39:48.439728 18517 ts_manager.cc:194] Registered new tserver with Master: 2ad3a4f015a54912885fa7587955aa71 (127.0.98.3:37067)
I20260924 04:39:48.440266 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53957
I20260924 04:39:48.450007 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:39:48.459188 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.451917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.0.98.62:43021)
0924 04:39:48.452059 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:48.452063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:48.452234 (+ 171us) server_negotiation.cc:407] Connection header received
0924 04:39:48.452328 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:48.452331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:48.452366 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:48.452429 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:48.453083 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.453581 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.454478 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.454676 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.454783 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.456182 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:48.456185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:48.456187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:48.456207 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:48.457670 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:48.458091 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:48.458093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:48.458094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:48.458162 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:48.458414 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:48.458416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:48.458417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:48.458589 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:48.458698 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:48.458903 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 04:39:48.459014 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:39:48.462026 18517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:39:48.464919 18517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:39:48.479437 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.472933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.0.98.1:44981)
0924 04:39:48.473894 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:48.473909 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:48.473931 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:39:48.474020 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:48.474025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:48.474233 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:48.474421 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:48.474752 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.475951 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.477812 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.478529 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.478625 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.478760 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:48.479193 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 04:39:48.479294 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":102,"threads_started":1}
I20260924 04:39:48.479465 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.472796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44981 (local address 127.0.0.1:48524)
0924 04:39:48.473185 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:48.473237 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:48.473338 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:48.474405 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:48.474411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:48.474435 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:48.474618 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.474627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.476077 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.476080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.477687 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.477698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.478633 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.478636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.478919 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.479024 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 04:39:48.479137 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 04:39:48.479289 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":376,"thread_start_us":95,"threads_started":1}
I20260924 04:39:48.479473 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.473060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34313 (local address 127.0.0.1:42364)
0924 04:39:48.473563 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:48.473577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:48.473658 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:48.474454 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:48.474457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:48.474474 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:48.474807 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.474815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.475888 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.475894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.477536 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.477546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.478633 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.478636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.478916 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.479024 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 04:39:48.479137 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 04:39:48.479289 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":465,"thread_start_us":112,"threads_started":1}
I20260924 04:39:48.479910 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.473578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.0.98.2:34313)
0924 04:39:48.473990 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:48.474004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:48.474025 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:39:48.474107 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:48.474115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:48.474298 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:48.474625 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:48.474925 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.475767 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.477659 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.478498 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.478616 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.478760 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:48.479680 (+ 920us) server_negotiation.cc:299] Negotiation successful
0924 04:39:48.479778 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":103,"threads_started":1}
I20260924 04:39:48.480183 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.473531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37067 (local address 127.0.0.1:49196)
0924 04:39:48.473939 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:48.473952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:48.474045 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:48.475140 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:48.475143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:48.475162 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:48.475233 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.475239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.476429 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.476432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.477688 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.477698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.479789 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.479793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.479880 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.479901 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:39:48.479975 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:39:48.480058 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":185,"thread_start_us":94,"threads_started":1}
I20260924 04:39:48.480605 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.473707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.0.98.3:37067)
0924 04:39:48.474721 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:48.474733 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:48.474753 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:39:48.474820 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:48.474824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:48.474998 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:48.475145 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:48.475342 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.476299 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.478816 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.479488 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.479596 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.479730 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:48.480344 (+ 614us) server_negotiation.cc:299] Negotiation successful
0924 04:39:48.480439 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":155,"threads_started":1}
I20260924 04:39:48.482637 18910 tablet_service.cc:1467] Processing CreateTablet for tablet 645c9391577d48ad8f9c70a49a406618 (DEFAULT_TABLE table=test-table [id=2c36ec3f90b845b88e6f1eec46a58d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:48.482646 18642 tablet_service.cc:1467] Processing CreateTablet for tablet 645c9391577d48ad8f9c70a49a406618 (DEFAULT_TABLE table=test-table [id=2c36ec3f90b845b88e6f1eec46a58d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:48.482857 18776 tablet_service.cc:1467] Processing CreateTablet for tablet 645c9391577d48ad8f9c70a49a406618 (DEFAULT_TABLE table=test-table [id=2c36ec3f90b845b88e6f1eec46a58d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:48.483592 18910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645c9391577d48ad8f9c70a49a406618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:48.483628 18776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645c9391577d48ad8f9c70a49a406618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:48.483628 18642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645c9391577d48ad8f9c70a49a406618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:48.489845 18993 tablet_bootstrap.cc:492] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: Bootstrap starting.
I20260924 04:39:48.491060 18995 tablet_bootstrap.cc:492] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: Bootstrap starting.
I20260924 04:39:48.491935 18993 tablet_bootstrap.cc:654] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:48.492756 18995 tablet_bootstrap.cc:654] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:48.492941 18993 log.cc:824] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:48.493468 18994 tablet_bootstrap.cc:492] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5: Bootstrap starting.
I20260924 04:39:48.494055 18995 log.cc:824] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:48.495113 18993 tablet_bootstrap.cc:492] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: No bootstrap required, opened a new log
I20260924 04:39:48.495309 18993 ts_tablet_manager.cc:1397] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:39:48.495530 18994 tablet_bootstrap.cc:654] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:48.496053 18995 tablet_bootstrap.cc:492] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: No bootstrap required, opened a new log
I20260924 04:39:48.496253 18995 ts_tablet_manager.cc:1397] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:39:48.496356 18994 log.cc:824] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:48.498288 18994 tablet_bootstrap.cc:492] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5: No bootstrap required, opened a new log
I20260924 04:39:48.498462 18994 ts_tablet_manager.cc:1397] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:39:48.498651 18993 raft_consensus.cc:348] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } }
I20260924 04:39:48.498896 18993 raft_consensus.cc:374] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:48.498947 18993 raft_consensus.cc:729] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad3a4f015a54912885fa7587955aa71, State: Initialized, Role: FOLLOWER
I20260924 04:39:48.499439 18993 consensus_queue.cc:260] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } }
I20260924 04:39:48.500360 18993 ts_tablet_manager.cc:1428] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:39:48.500419 18995 raft_consensus.cc:348] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } }
I20260924 04:39:48.500711 18976 heartbeater.cc:503] Master 127.0.98.62:43021 was elected leader, sending a full tablet report...
I20260924 04:39:48.500733 18995 raft_consensus.cc:374] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:48.500794 18995 raft_consensus.cc:729] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d07167221c0f4065aa5a3ecd02cd040d, State: Initialized, Role: FOLLOWER
I20260924 04:39:48.501210 18995 consensus_queue.cc:260] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } }
I20260924 04:39:48.501837 18842 heartbeater.cc:503] Master 127.0.98.62:43021 was elected leader, sending a full tablet report...
I20260924 04:39:48.502102 18995 ts_tablet_manager.cc:1428] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:39:48.502902 18994 raft_consensus.cc:348] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } }
I20260924 04:39:48.503111 18994 raft_consensus.cc:374] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:48.503152 18994 raft_consensus.cc:729] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcda3f8ee7314651bb9e25c602f807a5, State: Initialized, Role: FOLLOWER
I20260924 04:39:48.503515 18994 consensus_queue.cc:260] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } }
I20260924 04:39:48.504102 18708 heartbeater.cc:503] Master 127.0.98.62:43021 was elected leader, sending a full tablet report...
I20260924 04:39:48.504422 18994 ts_tablet_manager.cc:1428] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 04:39:48.674355 18977 tablet.cc:2369] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:39:48.704145 18999 raft_consensus.cc:482] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:48.704382 18999 raft_consensus.cc:504] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } }
I20260924 04:39:48.705603 18999 leader_election.cc:288] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcda3f8ee7314651bb9e25c602f807a5 (127.0.98.1:44981), d07167221c0f4065aa5a3ecd02cd040d (127.0.98.2:34313)
I20260924 04:39:48.709952 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.706219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34949 (local address 127.0.98.1:44981)
0924 04:39:48.706371 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:48.706375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:48.706595 (+ 220us) server_negotiation.cc:407] Connection header received
0924 04:39:48.706730 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:48.706734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:48.706785 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:48.706878 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:48.707092 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.707782 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.708925 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.709380 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.709515 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.709581 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:48.709776 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 04:39:48.709852 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:39:48.709950 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.706102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56359 (local address 127.0.98.2:34313)
0924 04:39:48.706242 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:48.706247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:48.706263 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:39:48.706467 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:48.706470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:48.706513 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:48.706604 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:48.706903 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.707704 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.708864 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.709355 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.709439 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.709527 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:48.709725 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 04:39:48.709813 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:39:48.709952 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.705994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34313 (local address 127.0.98.3:56359)
0924 04:39:48.706167 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:48.706184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:48.706296 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:48.706627 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:48.706631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:48.706645 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:48.706781 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.706786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.707801 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.707804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.708732 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.708742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.709479 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.709485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.709593 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.709619 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:39:48.709696 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:39:48.709807 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":11}
I20260924 04:39:48.709968 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.705994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44981 (local address 127.0.98.3:34949)
0924 04:39:48.706499 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:48.706509 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:48.706627 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:48.706888 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:48.706890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:48.706912 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:48.706994 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.706998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.707876 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.707882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.708816 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:48.708829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.709527 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.709532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:48.709636 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.709653 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:39:48.709749 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 04:39:48.709825 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":175,"thread_start_us":136,"threads_started":1}
I20260924 04:39:48.710599 18796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645c9391577d48ad8f9c70a49a406618" candidate_uuid: "2ad3a4f015a54912885fa7587955aa71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d07167221c0f4065aa5a3ecd02cd040d" is_pre_election: true
I20260924 04:39:48.710862 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645c9391577d48ad8f9c70a49a406618" candidate_uuid: "2ad3a4f015a54912885fa7587955aa71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcda3f8ee7314651bb9e25c602f807a5" is_pre_election: true
I20260924 04:39:48.710940 18796 raft_consensus.cc:2454] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad3a4f015a54912885fa7587955aa71 in term 0.
I20260924 04:39:48.711117 18662 raft_consensus.cc:2454] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad3a4f015a54912885fa7587955aa71 in term 0.
I20260924 04:39:48.711488 18864 leader_election.cc:302] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ad3a4f015a54912885fa7587955aa71, d07167221c0f4065aa5a3ecd02cd040d, dcda3f8ee7314651bb9e25c602f807a5; no voters:
I20260924 04:39:48.711817 18999 raft_consensus.cc:2781] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:39:48.711910 18999 raft_consensus.cc:482] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:39:48.711977 18999 raft_consensus.cc:3037] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:48.712992 18999 raft_consensus.cc:504] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } }
I20260924 04:39:48.713356 18999 leader_election.cc:288] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [CANDIDATE]: Term 1 election: Requested vote from peers dcda3f8ee7314651bb9e25c602f807a5 (127.0.98.1:44981), d07167221c0f4065aa5a3ecd02cd040d (127.0.98.2:34313)
I20260924 04:39:48.713711 18796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645c9391577d48ad8f9c70a49a406618" candidate_uuid: "2ad3a4f015a54912885fa7587955aa71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d07167221c0f4065aa5a3ecd02cd040d"
I20260924 04:39:48.713718 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645c9391577d48ad8f9c70a49a406618" candidate_uuid: "2ad3a4f015a54912885fa7587955aa71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcda3f8ee7314651bb9e25c602f807a5"
I20260924 04:39:48.713833 18662 raft_consensus.cc:3037] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:48.713833 18796 raft_consensus.cc:3037] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:48.715128 18662 raft_consensus.cc:2454] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad3a4f015a54912885fa7587955aa71 in term 1.
I20260924 04:39:48.715128 18796 raft_consensus.cc:2454] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad3a4f015a54912885fa7587955aa71 in term 1.
I20260924 04:39:48.715443 18864 leader_election.cc:302] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad3a4f015a54912885fa7587955aa71, d07167221c0f4065aa5a3ecd02cd040d; no voters:
I20260924 04:39:48.715631 18999 raft_consensus.cc:2781] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:48.715842 18999 raft_consensus.cc:686] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 1 LEADER]: Becoming Leader. State: Replica: 2ad3a4f015a54912885fa7587955aa71, State: Running, Role: LEADER
I20260924 04:39:48.716173 18999 consensus_queue.cc:237] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } }
W20260924 04:39:48.717429 18843 tablet.cc:2369] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:39:48.718400 18709 tablet.cc:2369] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:39:48.719363 18515 catalog_manager.cc:5495] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ad3a4f015a54912885fa7587955aa71 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2ad3a4f015a54912885fa7587955aa71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad3a4f015a54912885fa7587955aa71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 } health_report { overall_health: UNKNOWN } } }
I20260924 04:39:48.772567 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.769451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.0.98.3:37067)
0924 04:39:48.769587 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:48.769591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:48.769606 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:39:48.769695 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:48.769698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:48.769750 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:48.769850 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:39:48.770068 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.770555 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.771329 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:48.771535 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:48.771610 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:48.771850 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:39:48.772225 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:39:48.772328 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:48.772409 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 04:39:48.772443 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:39:48.778244 18796 raft_consensus.cc:1261] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [term 1 FOLLOWER]: Refusing update from remote peer 2ad3a4f015a54912885fa7587955aa71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:48.778244 18662 raft_consensus.cc:1261] T 645c9391577d48ad8f9c70a49a406618 P dcda3f8ee7314651bb9e25c602f807a5 [term 1 FOLLOWER]: Refusing update from remote peer 2ad3a4f015a54912885fa7587955aa71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:48.778914 18999 consensus_queue.cc:1035] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d07167221c0f4065aa5a3ecd02cd040d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:48.779109 19003 consensus_queue.cc:1035] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcda3f8ee7314651bb9e25c602f807a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:48.786583 19007 mvcc.cc:205] Tried to move back new op lower bound from 7332760734828679168 to 7332760734584008704. Current Snapshot: MvccSnapshot[applied={T|T < 7332760734828679168}]
I20260924 04:39:48.793732 18517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:48.793882 18517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:48.796172 18517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 072bcc849d1c4f1ab21bcd9b442225bf: Sending DeleteTablet for 3 replicas of tablet 645c9391577d48ad8f9c70a49a406618
I20260924 04:39:48.796944 18910 tablet_service.cc:1514] Processing DeleteTablet for tablet 645c9391577d48ad8f9c70a49a406618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49196
I20260924 04:39:48.796937 18776 tablet_service.cc:1514] Processing DeleteTablet for tablet 645c9391577d48ad8f9c70a49a406618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42364
I20260924 04:39:48.796978 18642 tablet_service.cc:1514] Processing DeleteTablet for tablet 645c9391577d48ad8f9c70a49a406618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48524
I20260924 04:39:48.797577 19016 tablet_replica.cc:330] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: stopping tablet replica
I20260924 04:39:48.797925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 18579
I20260924 04:39:48.799602 19018 tablet_replica.cc:330] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: stopping tablet replica
I20260924 04:39:48.799876 19018 raft_consensus.cc:2229] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:48.800196 19018 raft_consensus.cc:2258] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:48.800405 19016 raft_consensus.cc:2229] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:39:48.800712 19016 raft_consensus.cc:2258] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:48.801858 19016 ts_tablet_manager.cc:1905] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:48.801858 19018 ts_tablet_manager.cc:1905] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:48.804490 19018 ts_tablet_manager.cc:1918] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:48.804486 19016 ts_tablet_manager.cc:1918] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:48.804585 19018 log.cc:1196] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/645c9391577d48ad8f9c70a49a406618
I20260924 04:39:48.804699 19016 log.cc:1196] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/645c9391577d48ad8f9c70a49a406618
I20260924 04:39:48.804906 19018 ts_tablet_manager.cc:1939] T 645c9391577d48ad8f9c70a49a406618 P d07167221c0f4065aa5a3ecd02cd040d: Deleting consensus metadata
I20260924 04:39:48.805154 19016 ts_tablet_manager.cc:1939] T 645c9391577d48ad8f9c70a49a406618 P 2ad3a4f015a54912885fa7587955aa71: Deleting consensus metadata
I20260924 04:39:48.806084 18502 catalog_manager.cc:4841] TS d07167221c0f4065aa5a3ecd02cd040d (127.0.98.2:34313): tablet 645c9391577d48ad8f9c70a49a406618 (table test-table [id=2c36ec3f90b845b88e6f1eec46a58d8e]) successfully deleted
I20260924 04:39:48.806090 18504 catalog_manager.cc:4841] TS 2ad3a4f015a54912885fa7587955aa71 (127.0.98.3:37067): tablet 645c9391577d48ad8f9c70a49a406618 (table test-table [id=2c36ec3f90b845b88e6f1eec46a58d8e]) successfully deleted
W20260924 04:39:48.806572 18504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44981 (error 108)
I20260924 04:39:48.806746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 18713
I20260924 04:39:48.808326 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:48.808018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44981 (local address 127.0.0.1:48526)
0924 04:39:48.808148 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:48.808229 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 04:39:48.808488 18504 catalog_manager.cc:4584] TS dcda3f8ee7314651bb9e25c602f807a5 (127.0.98.1:44981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 645c9391577d48ad8f9c70a49a406618: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44981: connect: Connection refused (error 111)
I20260924 04:39:48.813832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 18847
I20260924 04:39:48.819187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 18485
2026-09-24T04:39:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2463 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T04:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:48Z Disabled control of system clock
I20260924 04:39:48.848155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43195
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:39:48.953907 19023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:48.954140 19023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:48.954174 19023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:48.958092 19023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:48.958142 19023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:48.958158 19023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:48.958174 19023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:48.958189 19023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:48.962757 19023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43195
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:48.963642 19023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:48.964852 19023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:48.972904 19028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:48.972883 19029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:48.972904 19031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:48.973373 19023 server_base.cc:1034] running on GCE node
I20260924 04:39:48.973930 19023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:48.974905 19023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:48.976056 19023 hybrid_clock.cc:648] HybridClock initialized: now 1790224788976029 us; error 40 us; skew 500 ppm
I20260924 04:39:48.977836 19023 webserver.cc:460] Webserver started at http://127.0.98.62:33343/ using document root <none> and password file <none>
I20260924 04:39:48.978370 19023 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:48.978430 19023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:48.978629 19023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:48.980544 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d"
format_stamp: "Formatted at 2026-09-24 04:39:48 on dist-test-slave-jlnt"
server_key: "8155097fa7590aaf444a79a573e8e084"
server_key_iv: "977de236d57f9b501618368200da6922"
server_key_version: "encryptionkey@0"
I20260924 04:39:48.981081 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d"
format_stamp: "Formatted at 2026-09-24 04:39:48 on dist-test-slave-jlnt"
server_key: "8155097fa7590aaf444a79a573e8e084"
server_key_iv: "977de236d57f9b501618368200da6922"
server_key_version: "encryptionkey@0"
I20260924 04:39:48.984541 19023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:39:48.987102 19037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:48.988561 19023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:39:48.988689 19023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d"
format_stamp: "Formatted at 2026-09-24 04:39:48 on dist-test-slave-jlnt"
server_key: "8155097fa7590aaf444a79a573e8e084"
server_key_iv: "977de236d57f9b501618368200da6922"
server_key_version: "encryptionkey@0"
I20260924 04:39:48.988801 19023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:49.046069 19023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:49.046801 19023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:49.047020 19023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:49.054028 19023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43195
I20260924 04:39:49.054056 19089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43195 every 8 connection(s)
I20260924 04:39:49.054970 19023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:49.058033 19090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:49.063428 19090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d: Bootstrap starting.
I20260924 04:39:49.064043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 19023
I20260924 04:39:49.064136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:49.064459 392 external_mini_cluster.cc:1428] Setting key ab7f23558d7320856e60538f59c2caae
I20260924 04:39:49.066318 19090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:49.067054 19090 log.cc:824] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:49.069209 19090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d: No bootstrap required, opened a new log
I20260924 04:39:49.073547 19090 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260924 04:39:49.073799 19090 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:49.073885 19090 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9fa5b8f2a8f4f53a1160ff1ee7e5d6d, State: Initialized, Role: FOLLOWER
I20260924 04:39:49.074000 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.065912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.0.98.62:43195)
0924 04:39:49.066618 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.066634 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.066671 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:39:49.067330 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.067348 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.067686 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.067975 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:49.068303 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.069468 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.070659 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.071121 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.071240 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.072095 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:49.072100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:49.072112 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:49.072485 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:49.072550 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:49.072569 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:49.072643 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.072827 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.073178 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":135,"threads_started":1}
I20260924 04:39:49.074447 19090 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260924 04:39:49.074604 19090 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:49.074666 19090 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:49.074754 19090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:49.075659 19090 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260924 04:39:49.076058 19090 leader_election.cc:302] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9fa5b8f2a8f4f53a1160ff1ee7e5d6d; no voters:
I20260924 04:39:49.076367 19090 leader_election.cc:288] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:49.076474 19095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:49.076725 19095 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [term 1 LEADER]: Becoming Leader. State: Replica: a9fa5b8f2a8f4f53a1160ff1ee7e5d6d, State: Running, Role: LEADER
I20260924 04:39:49.077029 19095 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260924 04:39:49.077670 19090 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:49.078729 19097 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9fa5b8f2a8f4f53a1160ff1ee7e5d6d. Latest consensus state: current_term: 1 leader_uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } } }
I20260924 04:39:49.078871 19097 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:49.078742 19096 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9fa5b8f2a8f4f53a1160ff1ee7e5d6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } } }
I20260924 04:39:49.078965 19096 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:49.079356 19104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:49.082103 19104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:49.087175 19104 catalog_manager.cc:1347] Generated new cluster ID: 8d39e40786524f40b8f89a68aaa54219
I20260924 04:39:49.087244 19104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:49.105835 19104 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:49.107125 19104 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:49.114820 19104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d: Generated new TSK 0
I20260924 04:39:49.115633 19104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:39:49.122411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--builtin_ntp_servers=127.0.98.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:39:49.232106 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:49.232373 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:49.232442 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:49.236342 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:49.236415 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:49.236502 19114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:39:49.241179 19114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:49.242134 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:49.243415 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:49.252583 19122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:49.252631 19120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:49.252604 19119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:49.252760 19114 server_base.cc:1034] running on GCE node
I20260924 04:39:49.253261 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:49.253834 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:49.255017 19114 hybrid_clock.cc:648] HybridClock initialized: now 1790224789254986 us; error 41 us; skew 500 ppm
I20260924 04:39:49.256844 19114 webserver.cc:460] Webserver started at http://127.0.98.1:41255/ using document root <none> and password file <none>
I20260924 04:39:49.257464 19114 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:49.257563 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:49.257761 19114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:49.259649 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "8accdd159f974e039beede1190d263b8"
format_stamp: "Formatted at 2026-09-24 04:39:49 on dist-test-slave-jlnt"
server_key: "6a4ecd989960c58f2c5de00540c6b640"
server_key_iv: "df8676b1f778144c9cae8b56b8af43c4"
server_key_version: "encryptionkey@0"
I20260924 04:39:49.260169 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8accdd159f974e039beede1190d263b8"
format_stamp: "Formatted at 2026-09-24 04:39:49 on dist-test-slave-jlnt"
server_key: "6a4ecd989960c58f2c5de00540c6b640"
server_key_iv: "df8676b1f778144c9cae8b56b8af43c4"
server_key_version: "encryptionkey@0"
I20260924 04:39:49.263545 19114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:39:49.265789 19128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:49.266928 19114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:49.267058 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "8accdd159f974e039beede1190d263b8"
format_stamp: "Formatted at 2026-09-24 04:39:49 on dist-test-slave-jlnt"
server_key: "6a4ecd989960c58f2c5de00540c6b640"
server_key_iv: "df8676b1f778144c9cae8b56b8af43c4"
server_key_version: "encryptionkey@0"
I20260924 04:39:49.267160 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:49.280316 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:49.280879 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:49.281054 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:49.281592 19114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:49.282440 19114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:49.282507 19114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:49.282568 19114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:49.282614 19114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:49.290910 19114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42563
I20260924 04:39:49.290934 19241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42563 every 8 connection(s)
I20260924 04:39:49.291782 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:39:49.298324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 19114
I20260924 04:39:49.298437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:39:49.298738 392 external_mini_cluster.cc:1428] Setting key 4064e7b2b34aefa50677ca2f6aec9c6a
I20260924 04:39:49.300868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--builtin_ntp_servers=127.0.98.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:39:49.300959 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.293505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59311 (local address 127.0.98.62:43195)
0924 04:39:49.293665 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.293670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.294668 (+ 998us) server_negotiation.cc:407] Connection header received
0924 04:39:49.295933 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.295936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.295982 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.296057 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:49.296846 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.297444 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.298892 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.299168 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.299284 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.300463 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:49.300467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:49.300470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:49.300482 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:49.300505 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:49.300515 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:49.300592 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.300745 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.300809 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:39:49.301714 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.293911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:59311)
0924 04:39:49.294528 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.294566 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:49.295750 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:49.296204 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:49.296210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:49.296288 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:49.296732 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.296744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.297583 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.297586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.298724 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.298734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.299330 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.299333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.299512 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.300200 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:39:49.300203 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:49.300251 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:39:49.300306 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:49.300605 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:49.300612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:49.300615 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:39:49.300703 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:39:49.300925 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260924 04:39:49.302950 19242 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43195
I20260924 04:39:49.303157 19242 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:49.303637 19242 heartbeater.cc:511] Master 127.0.98.62:43195 requested a full tablet report, sending...
I20260924 04:39:49.305050 19054 ts_manager.cc:194] Registered new tserver with Master: 8accdd159f974e039beede1190d263b8 (127.0.98.1:42563)
I20260924 04:39:49.306165 19054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59311
W20260924 04:39:49.408078 19245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:49.408303 19245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:49.408339 19245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:49.412307 19245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:49.412358 19245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:49.412451 19245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:39:49.417151 19245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:49.418102 19245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:49.419394 19245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:49.428579 19250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:49.428596 19251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:49.428745 19245 server_base.cc:1034] running on GCE node
W20260924 04:39:49.428586 19253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:49.429260 19245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:49.429885 19245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:49.431066 19245 hybrid_clock.cc:648] HybridClock initialized: now 1790224789431052 us; error 27 us; skew 500 ppm
I20260924 04:39:49.432775 19245 webserver.cc:460] Webserver started at http://127.0.98.2:34681/ using document root <none> and password file <none>
I20260924 04:39:49.433382 19245 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:49.433477 19245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:49.433693 19245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:49.435634 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "9066c65624e6464fbafae058d044f2f9"
format_stamp: "Formatted at 2026-09-24 04:39:49 on dist-test-slave-jlnt"
server_key: "74a7735d7a125bd498a102533d990a03"
server_key_iv: "ef54ed9e394bbb14cb61690ebc863a0d"
server_key_version: "encryptionkey@0"
I20260924 04:39:49.436177 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9066c65624e6464fbafae058d044f2f9"
format_stamp: "Formatted at 2026-09-24 04:39:49 on dist-test-slave-jlnt"
server_key: "74a7735d7a125bd498a102533d990a03"
server_key_iv: "ef54ed9e394bbb14cb61690ebc863a0d"
server_key_version: "encryptionkey@0"
I20260924 04:39:49.439604 19245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:39:49.441866 19259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:49.443005 19245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:49.443130 19245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "9066c65624e6464fbafae058d044f2f9"
format_stamp: "Formatted at 2026-09-24 04:39:49 on dist-test-slave-jlnt"
server_key: "74a7735d7a125bd498a102533d990a03"
server_key_iv: "ef54ed9e394bbb14cb61690ebc863a0d"
server_key_version: "encryptionkey@0"
I20260924 04:39:49.443230 19245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:49.457267 19245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:49.457875 19245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:49.458058 19245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:49.458592 19245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:49.459476 19245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:49.459553 19245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:49.459618 19245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:49.459659 19245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:49.468283 19245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37039
I20260924 04:39:49.468307 19372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37039 every 8 connection(s)
I20260924 04:39:49.469148 19245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:39:49.476838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 19245
I20260924 04:39:49.476974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:39:49.477367 392 external_mini_cluster.cc:1428] Setting key 5e8d5977503871feb28b287917b32029
I20260924 04:39:49.478473 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.471048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57255 (local address 127.0.98.62:43195)
0924 04:39:49.471194 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.471197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.472200 (+ 1003us) server_negotiation.cc:407] Connection header received
0924 04:39:49.473603 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.473606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.473651 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.473726 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:49.474552 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.475066 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.476127 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.476339 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.476416 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.477903 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:49.477906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:49.477908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:49.477920 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:49.477961 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:49.477971 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:49.478049 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.478251 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.478338 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:39:49.479413 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.471434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:57255)
0924 04:39:49.472080 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.472118 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:49.473424 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:49.473878 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:49.473884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:49.473964 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:49.474432 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.474444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.475168 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.475170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.476026 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.476034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.476500 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.476503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.476767 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.477592 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:39:49.477595 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:49.477648 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:39:49.477724 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:49.478074 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:49.478083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:49.478089 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:39:49.478243 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 04:39:49.478544 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260924 04:39:49.479447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--builtin_ntp_servers=127.0.98.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:39:49.480562 19373 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43195
I20260924 04:39:49.480811 19373 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:49.481374 19373 heartbeater.cc:511] Master 127.0.98.62:43195 requested a full tablet report, sending...
I20260924 04:39:49.482318 19054 ts_manager.cc:194] Registered new tserver with Master: 9066c65624e6464fbafae058d044f2f9 (127.0.98.2:37039)
I20260924 04:39:49.482877 19054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57255
W20260924 04:39:49.587229 19376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:49.587499 19376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:49.587569 19376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:49.591593 19376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:49.591673 19376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:49.591755 19376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:39:49.596486 19376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:49.597605 19376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:49.598892 19376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:49.607792 19384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:49.607936 19382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:49.607946 19381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:49.608203 19376 server_base.cc:1034] running on GCE node
I20260924 04:39:49.608585 19376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:49.609231 19376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:49.610416 19376 hybrid_clock.cc:648] HybridClock initialized: now 1790224789610393 us; error 36 us; skew 500 ppm
I20260924 04:39:49.612161 19376 webserver.cc:460] Webserver started at http://127.0.98.3:37801/ using document root <none> and password file <none>
I20260924 04:39:49.612723 19376 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:49.612811 19376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:49.613036 19376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:49.615137 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "a1826b5a8a0c44668370d3205b68ae5d"
format_stamp: "Formatted at 2026-09-24 04:39:49 on dist-test-slave-jlnt"
server_key: "369a47cb93ff0359c0f8f6150852a8ec"
server_key_iv: "4594a6eac8ea11ea4d9e38066382a227"
server_key_version: "encryptionkey@0"
I20260924 04:39:49.615684 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a1826b5a8a0c44668370d3205b68ae5d"
format_stamp: "Formatted at 2026-09-24 04:39:49 on dist-test-slave-jlnt"
server_key: "369a47cb93ff0359c0f8f6150852a8ec"
server_key_iv: "4594a6eac8ea11ea4d9e38066382a227"
server_key_version: "encryptionkey@0"
I20260924 04:39:49.619174 19376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:39:49.621419 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:49.622568 19376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:39:49.622696 19376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "a1826b5a8a0c44668370d3205b68ae5d"
format_stamp: "Formatted at 2026-09-24 04:39:49 on dist-test-slave-jlnt"
server_key: "369a47cb93ff0359c0f8f6150852a8ec"
server_key_iv: "4594a6eac8ea11ea4d9e38066382a227"
server_key_version: "encryptionkey@0"
I20260924 04:39:49.622802 19376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:49.642964 19376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:49.643564 19376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:49.643747 19376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:49.644275 19376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:49.645125 19376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:49.645244 19376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:49.645308 19376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:49.645355 19376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:49.653838 19376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33715
I20260924 04:39:49.653867 19503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33715 every 8 connection(s)
I20260924 04:39:49.654709 19376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:39:49.655786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 19376
I20260924 04:39:49.655900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:39:49.656194 392 external_mini_cluster.cc:1428] Setting key 1cb06de1b9d52973ead2dc3f227882c6
I20260924 04:39:49.663273 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.656525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35333 (local address 127.0.98.62:43195)
0924 04:39:49.656682 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.656686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.657898 (+ 1212us) server_negotiation.cc:407] Connection header received
0924 04:39:49.659256 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.659259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.659301 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.659370 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:49.660126 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.660626 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.661730 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.661948 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.662024 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.662918 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:49.662921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:49.662922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:49.662933 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:49.662950 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:49.662957 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:49.663013 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.663130 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.663172 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:39:49.664016 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.656997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:35333)
0924 04:39:49.657749 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.657796 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:49.659076 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:49.659490 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:49.659495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:49.659577 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:49.660010 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.660021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.660722 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.660724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.661632 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.661640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.662062 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.662064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.662197 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.662702 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:39:49.662704 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:49.662738 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:39:49.662794 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:49.663027 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:49.663031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:49.663034 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:39:49.663114 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:39:49.663304 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":182,"threads_started":1}
I20260924 04:39:49.664981 19504 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43195
I20260924 04:39:49.665212 19504 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:49.665622 19504 heartbeater.cc:511] Master 127.0.98.62:43195 requested a full tablet report, sending...
I20260924 04:39:49.666468 19054 ts_manager.cc:194] Registered new tserver with Master: a1826b5a8a0c44668370d3205b68ae5d (127.0.98.3:33715)
I20260924 04:39:49.667001 19054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35333
I20260924 04:39:49.669826 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:39:49.675318 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.671773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.0.98.62:43195)
0924 04:39:49.671885 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.671888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.672089 (+ 201us) server_negotiation.cc:407] Connection header received
0924 04:39:49.672175 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.672176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.672210 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.672271 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:49.672539 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.673047 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.673893 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.674092 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.674172 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.674941 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:49.674944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:49.674945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:49.674955 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:49.674971 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:49.674978 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:49.675073 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.675195 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.675224 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:39:49.677999 19054 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:39:49.681006 19054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:39:49.695178 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.688897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42563 (local address 127.0.0.1:33270)
0924 04:39:49.689828 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.689843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:49.689935 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:49.690481 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:49.690486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:49.690506 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:49.690730 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.690738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.692038 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.692041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.693651 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.693662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.694646 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.694650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.694753 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.694795 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 04:39:49.694913 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:39:49.695040 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":148,"spinlock_wait_cycles":384,"thread_start_us":76,"threads_started":1}
I20260924 04:39:49.695200 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.689269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.0.98.1:42563)
0924 04:39:49.689668 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.689682 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.689973 (+ 291us) server_negotiation.cc:407] Connection header received
0924 04:39:49.690045 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.690049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.690193 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.690319 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:49.690867 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.691948 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.693792 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.694512 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.694595 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.694722 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.694955 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.695050 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":132,"threads_started":1}
I20260924 04:39:49.695686 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.688542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37039 (local address 127.0.0.1:59276)
0924 04:39:49.689197 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.689246 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:49.690023 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:49.690506 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:49.690508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:49.690520 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:49.690735 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.690739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.693187 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.693192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.694158 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.694169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.695271 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.695276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.695379 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.695399 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:39:49.695471 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:39:49.695535 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":56,"spinlock_wait_cycles":1540480,"thread_start_us":73,"threads_started":1}
I20260924 04:39:49.696293 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.688619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33715 (local address 127.0.0.1:50064)
0924 04:39:49.689197 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.689246 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:49.689767 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:49.690313 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:49.690318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:49.690339 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:49.690474 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.690484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.691904 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.691907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.693652 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.693662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.695887 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.695890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.695997 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.696029 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 04:39:49.696093 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:39:49.696159 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":83,"spinlock_wait_cycles":4352,"thread_start_us":91,"threads_started":1}
I20260924 04:39:49.696363 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.688670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.0.98.2:37039)
0924 04:39:49.689630 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.689644 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.689667 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:39:49.690162 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.690167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.690373 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.690528 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:49.692055 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.693064 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.694288 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.694954 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.695045 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.695204 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.696116 (+ 912us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.696206 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":151,"threads_started":1}
I20260924 04:39:49.697356 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.689247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.0.98.3:33715)
0924 04:39:49.689741 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.689752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.689774 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:39:49.689956 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.689962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.690143 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.690314 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:49.690601 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.691770 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.694777 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.695543 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.695643 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.695830 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.697102 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.697209 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":138,"threads_started":1}
I20260924 04:39:49.698405 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 83d158d2997749d8806ce7653b240714 (DEFAULT_TABLE table=test-table [id=9f9de8c28c454afbab2cd75194380efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:49.698859 19307 tablet_service.cc:1467] Processing CreateTablet for tablet 83d158d2997749d8806ce7653b240714 (DEFAULT_TABLE table=test-table [id=9f9de8c28c454afbab2cd75194380efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:49.699338 19176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d158d2997749d8806ce7653b240714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:49.699847 19307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d158d2997749d8806ce7653b240714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:49.699934 19438 tablet_service.cc:1467] Processing CreateTablet for tablet 83d158d2997749d8806ce7653b240714 (DEFAULT_TABLE table=test-table [id=9f9de8c28c454afbab2cd75194380efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:49.700788 19438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d158d2997749d8806ce7653b240714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:49.705325 19521 tablet_bootstrap.cc:492] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8: Bootstrap starting.
I20260924 04:39:49.706686 19522 tablet_bootstrap.cc:492] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: Bootstrap starting.
I20260924 04:39:49.707525 19523 tablet_bootstrap.cc:492] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: Bootstrap starting.
I20260924 04:39:49.707791 19521 tablet_bootstrap.cc:654] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:49.708675 19521 log.cc:824] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:49.708757 19522 tablet_bootstrap.cc:654] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:49.709340 19523 tablet_bootstrap.cc:654] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:49.709568 19522 log.cc:824] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:49.710136 19523 log.cc:824] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:49.710820 19521 tablet_bootstrap.cc:492] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8: No bootstrap required, opened a new log
I20260924 04:39:49.711046 19521 ts_tablet_manager.cc:1397] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:39:49.711475 19522 tablet_bootstrap.cc:492] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: No bootstrap required, opened a new log
I20260924 04:39:49.711660 19522 ts_tablet_manager.cc:1397] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:39:49.711880 19523 tablet_bootstrap.cc:492] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: No bootstrap required, opened a new log
I20260924 04:39:49.712074 19523 ts_tablet_manager.cc:1397] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:39:49.715102 19523 raft_consensus.cc:348] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } }
I20260924 04:39:49.715529 19523 raft_consensus.cc:374] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:49.715646 19523 raft_consensus.cc:729] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1826b5a8a0c44668370d3205b68ae5d, State: Initialized, Role: FOLLOWER
I20260924 04:39:49.715472 19521 raft_consensus.cc:348] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } }
I20260924 04:39:49.715744 19521 raft_consensus.cc:374] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:49.715865 19521 raft_consensus.cc:729] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8accdd159f974e039beede1190d263b8, State: Initialized, Role: FOLLOWER
I20260924 04:39:49.715780 19522 raft_consensus.cc:348] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } }
I20260924 04:39:49.716015 19522 raft_consensus.cc:374] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:49.716068 19522 raft_consensus.cc:729] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9066c65624e6464fbafae058d044f2f9, State: Initialized, Role: FOLLOWER
I20260924 04:39:49.716221 19523 consensus_queue.cc:260] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } }
I20260924 04:39:49.716320 19521 consensus_queue.cc:260] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } }
I20260924 04:39:49.716501 19522 consensus_queue.cc:260] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } }
I20260924 04:39:49.717357 19522 ts_tablet_manager.cc:1428] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:39:49.717356 19521 ts_tablet_manager.cc:1428] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:39:49.717356 19523 ts_tablet_manager.cc:1428] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:39:49.717736 19242 heartbeater.cc:503] Master 127.0.98.62:43195 was elected leader, sending a full tablet report...
I20260924 04:39:49.717914 19504 heartbeater.cc:503] Master 127.0.98.62:43195 was elected leader, sending a full tablet report...
I20260924 04:39:49.718660 19373 heartbeater.cc:503] Master 127.0.98.62:43195 was elected leader, sending a full tablet report...
W20260924 04:39:49.720068 19374 tablet.cc:2369] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:39:49.734510 19527 raft_consensus.cc:482] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:49.734772 19527 raft_consensus.cc:504] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } }
I20260924 04:39:49.735924 19527 leader_election.cc:288] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9066c65624e6464fbafae058d044f2f9 (127.0.98.2:37039), 8accdd159f974e039beede1190d263b8 (127.0.98.1:42563)
I20260924 04:39:49.739658 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.736086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42563 (local address 127.0.98.3:34959)
0924 04:39:49.736255 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.736271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:49.736357 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:49.736623 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:49.736625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:49.736642 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:49.736781 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.736787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.737664 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.737667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.738514 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.738522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.739192 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.739196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.739359 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.739388 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:39:49.739455 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:39:49.739545 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":30}
I20260924 04:39:49.739694 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.736204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34959 (local address 127.0.98.1:42563)
0924 04:39:49.736335 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.736341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.736356 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:39:49.736471 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.736474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.736520 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.736601 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:49.736907 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.737554 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.738640 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.739070 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.739147 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.739192 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.739499 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.739577 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 04:39:49.739986 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.736192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56507 (local address 127.0.98.2:37039)
0924 04:39:49.736320 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.736324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.736697 (+ 373us) server_negotiation.cc:407] Connection header received
0924 04:39:49.736789 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.736793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.736849 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.736920 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:49.737225 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.737753 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.738760 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.739188 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.739288 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.739360 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.739779 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.739884 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:39:49.740296 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.736086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37039 (local address 127.0.98.3:56507)
0924 04:39:49.736601 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.736615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:49.736706 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:49.736942 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:49.736944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:49.736961 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:49.737075 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.737082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.737854 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.737856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.738646 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:49.738656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.739301 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.739303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:49.739408 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.739425 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:39:49.740138 (+ 713us) client_negotiation.cc:240] Negotiation successful
0924 04:39:49.740200 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":173,"thread_start_us":139,"threads_started":1}
I20260924 04:39:49.740307 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d158d2997749d8806ce7653b240714" candidate_uuid: "a1826b5a8a0c44668370d3205b68ae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8accdd159f974e039beede1190d263b8" is_pre_election: true
I20260924 04:39:49.740595 19196 raft_consensus.cc:2454] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1826b5a8a0c44668370d3205b68ae5d in term 0.
I20260924 04:39:49.740868 19327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d158d2997749d8806ce7653b240714" candidate_uuid: "a1826b5a8a0c44668370d3205b68ae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9066c65624e6464fbafae058d044f2f9" is_pre_election: true
I20260924 04:39:49.741117 19327 raft_consensus.cc:2454] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1826b5a8a0c44668370d3205b68ae5d in term 0.
I20260924 04:39:49.741077 19394 leader_election.cc:302] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8accdd159f974e039beede1190d263b8, a1826b5a8a0c44668370d3205b68ae5d; no voters:
I20260924 04:39:49.741366 19527 raft_consensus.cc:2781] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:39:49.741473 19527 raft_consensus.cc:482] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:39:49.741526 19527 raft_consensus.cc:3037] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:49.742484 19527 raft_consensus.cc:504] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } }
I20260924 04:39:49.742798 19527 leader_election.cc:288] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [CANDIDATE]: Term 1 election: Requested vote from peers 9066c65624e6464fbafae058d044f2f9 (127.0.98.2:37039), 8accdd159f974e039beede1190d263b8 (127.0.98.1:42563)
I20260924 04:39:49.743074 19327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d158d2997749d8806ce7653b240714" candidate_uuid: "a1826b5a8a0c44668370d3205b68ae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9066c65624e6464fbafae058d044f2f9"
I20260924 04:39:49.743131 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d158d2997749d8806ce7653b240714" candidate_uuid: "a1826b5a8a0c44668370d3205b68ae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8accdd159f974e039beede1190d263b8"
I20260924 04:39:49.743211 19327 raft_consensus.cc:3037] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:49.743228 19196 raft_consensus.cc:3037] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:49.744419 19327 raft_consensus.cc:2454] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1826b5a8a0c44668370d3205b68ae5d in term 1.
I20260924 04:39:49.744423 19196 raft_consensus.cc:2454] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1826b5a8a0c44668370d3205b68ae5d in term 1.
I20260924 04:39:49.744729 19392 leader_election.cc:302] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9066c65624e6464fbafae058d044f2f9, a1826b5a8a0c44668370d3205b68ae5d; no voters:
I20260924 04:39:49.744916 19527 raft_consensus.cc:2781] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:49.745105 19527 raft_consensus.cc:686] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 1 LEADER]: Becoming Leader. State: Replica: a1826b5a8a0c44668370d3205b68ae5d, State: Running, Role: LEADER
I20260924 04:39:49.745451 19527 consensus_queue.cc:237] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } }
I20260924 04:39:49.748632 19052 catalog_manager.cc:5495] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d reported cstate change: term changed from 0 to 1, leader changed from <none> to a1826b5a8a0c44668370d3205b68ae5d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a1826b5a8a0c44668370d3205b68ae5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1826b5a8a0c44668370d3205b68ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 } health_report { overall_health: UNKNOWN } } }
I20260924 04:39:49.772181 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.768343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.0.98.3:33715)
0924 04:39:49.768494 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:49.768498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:49.768513 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:39:49.768644 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:49.768646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:49.768698 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:49.768806 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:39:49.769053 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.769612 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.770712 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:49.770947 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:49.771066 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:49.771374 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:39:49.771750 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:39:49.771915 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:49.771999 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:39:49.772032 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 04:39:49.777741 19196 raft_consensus.cc:1261] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [term 1 FOLLOWER]: Refusing update from remote peer a1826b5a8a0c44668370d3205b68ae5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:49.777741 19327 raft_consensus.cc:1261] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [term 1 FOLLOWER]: Refusing update from remote peer a1826b5a8a0c44668370d3205b68ae5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:49.778439 19527 consensus_queue.cc:1035] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9066c65624e6464fbafae058d044f2f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:49.778600 19531 consensus_queue.cc:1035] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8accdd159f974e039beede1190d263b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:49.784394 19534 mvcc.cc:205] Tried to move back new op lower bound from 7332760738922995712 to 7332760738800222208. Current Snapshot: MvccSnapshot[applied={T|T < 7332760738922995712}]
I20260924 04:39:49.785588 19536 mvcc.cc:205] Tried to move back new op lower bound from 7332760738922995712 to 7332760738800222208. Current Snapshot: MvccSnapshot[applied={T|T < 7332760738922995712}]
I20260924 04:39:49.789001 19535 mvcc.cc:205] Tried to move back new op lower bound from 7332760738922995712 to 7332760738800222208. Current Snapshot: MvccSnapshot[applied={T|T < 7332760738922995712}]
I20260924 04:39:49.793051 19053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:49.793224 19053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:49.795466 19053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9fa5b8f2a8f4f53a1160ff1ee7e5d6d: Sending DeleteTablet for 3 replicas of tablet 83d158d2997749d8806ce7653b240714
I20260924 04:39:49.796126 19307 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d158d2997749d8806ce7653b240714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:49 UTC) from {username='slave'} at 127.0.0.1:59276
I20260924 04:39:49.796227 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d158d2997749d8806ce7653b240714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:49 UTC) from {username='slave'} at 127.0.0.1:33270
I20260924 04:39:49.796574 19544 tablet_replica.cc:330] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: stopping tablet replica
I20260924 04:39:49.796579 19438 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d158d2997749d8806ce7653b240714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:49 UTC) from {username='slave'} at 127.0.0.1:50064
I20260924 04:39:49.796660 19545 tablet_replica.cc:330] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8: stopping tablet replica
I20260924 04:39:49.796919 19545 raft_consensus.cc:2229] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:49.796921 19544 raft_consensus.cc:2229] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:49.797221 19544 raft_consensus.cc:2258] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:49.797221 19545 raft_consensus.cc:2258] T 83d158d2997749d8806ce7653b240714 P 8accdd159f974e039beede1190d263b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:49.797741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 19114
I20260924 04:39:49.802619 19546 tablet_replica.cc:330] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: stopping tablet replica
I20260924 04:39:49.802956 19546 raft_consensus.cc:2229] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:39:49.803321 19546 raft_consensus.cc:2258] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:49.803345 19544 ts_tablet_manager.cc:1905] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:39:49.805054 19041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42563 (error 108)
I20260924 04:39:49.805603 19546 ts_tablet_manager.cc:1905] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:49.805785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 19245
I20260924 04:39:49.806133 19544 ts_tablet_manager.cc:1918] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:49.806207 19544 log.cc:1196] T 83d158d2997749d8806ce7653b240714 P 9066c65624e6464fbafae058d044f2f9: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/83d158d2997749d8806ce7653b240714
I20260924 04:39:49.808882 19546 ts_tablet_manager.cc:1918] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:49.808972 19546 log.cc:1196] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/83d158d2997749d8806ce7653b240714
I20260924 04:39:49.809278 19546 ts_tablet_manager.cc:1939] T 83d158d2997749d8806ce7653b240714 P a1826b5a8a0c44668370d3205b68ae5d: Deleting consensus metadata
I20260924 04:39:49.810469 19040 catalog_manager.cc:4841] TS a1826b5a8a0c44668370d3205b68ae5d (127.0.98.3:33715): tablet 83d158d2997749d8806ce7653b240714 (table test-table [id=9f9de8c28c454afbab2cd75194380efe]) successfully deleted
I20260924 04:39:49.811026 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.810722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42563 (local address 127.0.0.1:33278)
0924 04:39:49.810852 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.810929 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 04:39:49.811195 19041 catalog_manager.cc:4584] TS 8accdd159f974e039beede1190d263b8 (127.0.98.1:42563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83d158d2997749d8806ce7653b240714: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42563: connect: Connection refused (error 111)
I20260924 04:39:49.812465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 19376
I20260924 04:39:49.813417 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:49.812624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37039 (local address 127.0.0.1:59278)
0924 04:39:49.813267 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:49.813327 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551}
I20260924 04:39:49.818123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 19023
2026-09-24T04:39:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (998 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:39:49 dist-test-slave-jlnt krb5kdc[19550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:39:49 dist-test-slave-jlnt krb5kdc[19550](info): set up 2 sockets
Sep 24 04:39:49 dist-test-slave-jlnt krb5kdc[19550](info): commencing operation
krb5kdc: starting...
W20260924 04:39:50.845078 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:39:50 dist-test-slave-jlnt krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:39:50.964176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36407
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:51.071748 19563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:51.072017 19563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:51.072058 19563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:51.076259 19563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:51.076323 19563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:51.076341 19563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:51.076359 19563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:51.076375 19563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:51.081722 19563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:51.082729 19563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:51.083719 19563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:51.092031 19571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:51.092160 19568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:51.092108 19563 server_base.cc:1034] running on GCE node
W20260924 04:39:51.092048 19569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:51.092912 19563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:51.093981 19563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:51.095149 19563 hybrid_clock.cc:648] HybridClock initialized: now 1790224791095136 us; error 24 us; skew 500 ppm
Sep 24 04:39:51 dist-test-slave-jlnt krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:51.098095 19563 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:39:51.098939 19563 webserver.cc:460] Webserver started at http://127.0.98.62:34197/ using document root <none> and password file <none>
I20260924 04:39:51.099516 19563 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:51.099618 19563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:51.099843 19563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:51.101863 19563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "d03ab1939bfc468a9d5473531228e6a3"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "cd45f3f4d04793887a85d1d789d8b719"
server_key_iv: "c5b2ca29f5b43ca9cf712559fe67b321"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.102430 19563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "d03ab1939bfc468a9d5473531228e6a3"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "cd45f3f4d04793887a85d1d789d8b719"
server_key_iv: "c5b2ca29f5b43ca9cf712559fe67b321"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.105957 19563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:39:51.108248 19578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.109423 19563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:51.109552 19563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "d03ab1939bfc468a9d5473531228e6a3"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "cd45f3f4d04793887a85d1d789d8b719"
server_key_iv: "c5b2ca29f5b43ca9cf712559fe67b321"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.109658 19563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:51.127837 19563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:51.131043 19563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:51.131243 19563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:51.137781 19563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36407
I20260924 04:39:51.137811 19630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36407 every 8 connection(s)
I20260924 04:39:51.138654 19563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:51.139820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 19563
I20260924 04:39:51.139954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:51.140272 392 external_mini_cluster.cc:1428] Setting key e76fd9defa6db9a250affbfda3f29d33
I20260924 04:39:51.141697 19631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:51.146903 19631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3: Bootstrap starting.
Sep 24 04:39:51 dist-test-slave-jlnt krb5kdc[19550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:51.149261 19631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:51.149956 19631 log.cc:824] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:51.151716 19631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3: No bootstrap required, opened a new log
I20260924 04:39:51.153753 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.141634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.0.98.62:36407)
0924 04:39:51.142031 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:51.142048 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:51.142083 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:39:51.142714 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:51.142738 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:51.143128 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:51.143450 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:51.144229 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.145254 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.146089 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.146563 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.146726 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.149346 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:51.149354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:51.149370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:51.149404 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:51.151127 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.151566 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.151576 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.151580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.151684 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.151979 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.151985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.151989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.152346 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:51.152558 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:51.152780 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 04:39:51.153045 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":139,"threads_started":1}
I20260924 04:39:51.155207 19631 raft_consensus.cc:348] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03ab1939bfc468a9d5473531228e6a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36407 } }
I20260924 04:39:51.155447 19631 raft_consensus.cc:374] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:51.155505 19631 raft_consensus.cc:729] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03ab1939bfc468a9d5473531228e6a3, State: Initialized, Role: FOLLOWER
I20260924 04:39:51.155962 19631 consensus_queue.cc:260] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03ab1939bfc468a9d5473531228e6a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36407 } }
I20260924 04:39:51.156116 19631 raft_consensus.cc:388] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:51.156203 19631 raft_consensus.cc:482] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:51.156324 19631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:51.157281 19631 raft_consensus.cc:504] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03ab1939bfc468a9d5473531228e6a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36407 } }
I20260924 04:39:51.157605 19631 leader_election.cc:302] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d03ab1939bfc468a9d5473531228e6a3; no voters:
I20260924 04:39:51.157895 19631 leader_election.cc:288] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:51.158010 19636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:51.158246 19636 raft_consensus.cc:686] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [term 1 LEADER]: Becoming Leader. State: Replica: d03ab1939bfc468a9d5473531228e6a3, State: Running, Role: LEADER
I20260924 04:39:51.158540 19636 consensus_queue.cc:237] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03ab1939bfc468a9d5473531228e6a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36407 } }
I20260924 04:39:51.159107 19631 sys_catalog.cc:564] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:51.159667 19636 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03ab1939bfc468a9d5473531228e6a3. Latest consensus state: current_term: 1 leader_uuid: "d03ab1939bfc468a9d5473531228e6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03ab1939bfc468a9d5473531228e6a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36407 } } }
I20260924 04:39:51.159782 19636 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:51.160162 19642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:51.160104 19637 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d03ab1939bfc468a9d5473531228e6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03ab1939bfc468a9d5473531228e6a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36407 } } }
I20260924 04:39:51.160274 19637 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:51.162768 19642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:51.167946 19642 catalog_manager.cc:1347] Generated new cluster ID: da6523c993ee413e91f037b10a1faf7f
I20260924 04:39:51.168005 19642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:51.203493 19642 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:51.204754 19642 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:51.220993 19642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3: Generated new TSK 0
I20260924 04:39:51.221730 19642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:39:51.254441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36407
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:51.362007 19657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:51.362257 19657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:51.362323 19657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:51.366269 19657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:51.366340 19657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:51.366427 19657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:39:51.372368 19657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:51.373706 19657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:51.374704 19657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:51.383875 19663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:51.383878 19665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:51.383891 19662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:51.384275 19657 server_base.cc:1034] running on GCE node
I20260924 04:39:51.384666 19657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:51.385317 19657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:51.386509 19657 hybrid_clock.cc:648] HybridClock initialized: now 1790224791386480 us; error 45 us; skew 500 ppm
Sep 24 04:39:51 dist-test-slave-jlnt krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:51.389108 19657 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:39:51.389950 19657 webserver.cc:460] Webserver started at http://127.0.98.1:43593/ using document root <none> and password file <none>
I20260924 04:39:51.390441 19657 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:51.390506 19657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:51.390661 19657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:51.392549 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "2516244cfd56188a448a87e63c7b3287"
server_key_iv: "d763f95a1739aef7c0cf9963cfc3523d"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.393029 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "2516244cfd56188a448a87e63c7b3287"
server_key_iv: "d763f95a1739aef7c0cf9963cfc3523d"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.396445 19657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:39:51.398749 19672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.399904 19657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:39:51.400002 19657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "2516244cfd56188a448a87e63c7b3287"
server_key_iv: "d763f95a1739aef7c0cf9963cfc3523d"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.400077 19657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:51.418545 19657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:51.421532 19657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:51.421679 19657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:51.422164 19657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:51.422894 19657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:51.422940 19657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.422978 19657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:51.422995 19657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.431766 19657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35483
I20260924 04:39:51.431789 19785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35483 every 8 connection(s)
I20260924 04:39:51.432657 19657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:39:51.440349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 19657
I20260924 04:39:51.440505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:39:51.440773 392 external_mini_cluster.cc:1428] Setting key 0f3c0e66d77c32a06ea0adcc165118ad
Sep 24 04:39:51 dist-test-slave-jlnt krb5kdc[19550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:51.445762 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.434425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34425 (local address 127.0.98.62:36407)
0924 04:39:51.434561 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:51.434565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:51.435349 (+ 784us) server_negotiation.cc:407] Connection header received
0924 04:39:51.436526 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:51.436529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:51.436575 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:51.436648 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:51.437773 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.438357 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.439408 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.439636 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.439714 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.442177 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:51.442181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:51.442184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:51.442211 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:51.443609 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.444164 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.444170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.444173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.444269 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.444577 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.444583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.444586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.444827 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:51.444934 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:51.445549 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 04:39:51.445648 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:39:51.446913 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.434786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36407 (local address 127.0.98.1:34425)
0924 04:39:51.435193 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:51.435234 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:51.436319 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:51.436809 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:51.436815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:51.437185 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:51.437646 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.437659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.438464 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.438466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.439304 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.439312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.439757 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.439760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.439890 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.440498 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:51.440517 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:51.442029 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:51.443743 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:51.443750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:51.443768 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:51.444061 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:51.444361 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:51.444363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:51.444364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:51.444488 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:51.444969 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:51.444978 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:51.445294 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 04:39:51.445492 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 04:39:51.445821 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260924 04:39:51.448061 19786 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36407
I20260924 04:39:51.448276 19786 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:51.448731 19786 heartbeater.cc:511] Master 127.0.98.62:36407 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:39:51.450265 19595 ts_manager.cc:194] Registered new tserver with Master: e5e1eb7f1dc34e05a36db6f43d602b1a (127.0.98.1:35483)
I20260924 04:39:51.451381 19595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:39:51.468278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36407
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:51.576107 19791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:51.576342 19791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:51.576385 19791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:51.580206 19791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:51.580251 19791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:51.580344 19791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:39:51.585391 19791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:51.586295 19791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:51.587221 19791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:51.596329 19796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:51.596329 19797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:51.596485 19791 server_base.cc:1034] running on GCE node
W20260924 04:39:51.596427 19799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:51.597014 19791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:51.597601 19791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:51.598788 19791 hybrid_clock.cc:648] HybridClock initialized: now 1790224791598763 us; error 41 us; skew 500 ppm
Sep 24 04:39:51 dist-test-slave-jlnt krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:51.601496 19791 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:39:51.602334 19791 webserver.cc:460] Webserver started at http://127.0.98.2:41805/ using document root <none> and password file <none>
I20260924 04:39:51.602880 19791 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:51.602936 19791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:51.603086 19791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:51.605015 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "8225129f8b50490581a5cae3deb6e49d"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "1d0240b460d22cf42e1c6f2e2273a740"
server_key_iv: "e564330cca6ed4171c6a9dde493f3d1b"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.605557 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8225129f8b50490581a5cae3deb6e49d"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "1d0240b460d22cf42e1c6f2e2273a740"
server_key_iv: "e564330cca6ed4171c6a9dde493f3d1b"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.608919 19791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:39:51.611241 19806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.612380 19791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:51.612494 19791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "8225129f8b50490581a5cae3deb6e49d"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "1d0240b460d22cf42e1c6f2e2273a740"
server_key_iv: "e564330cca6ed4171c6a9dde493f3d1b"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.612586 19791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:51.623059 19791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:51.626231 19791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:51.626367 19791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:51.626834 19791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:51.627511 19791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:51.627566 19791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.627602 19791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:51.627619 19791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.635993 19791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45897
I20260924 04:39:51.636027 19919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45897 every 8 connection(s)
I20260924 04:39:51.636794 19791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:39:51.643857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 19791
I20260924 04:39:51.643965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:39:51.644258 392 external_mini_cluster.cc:1428] Setting key 37286a9e4af806de0436450408598d6a
Sep 24 04:39:51 dist-test-slave-jlnt krb5kdc[19550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:51.650646 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.638502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39349 (local address 127.0.98.62:36407)
0924 04:39:51.638664 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:51.638667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:51.639409 (+ 742us) server_negotiation.cc:407] Connection header received
0924 04:39:51.640616 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:51.640619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:51.640664 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:51.640735 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:51.641876 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.642384 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.643425 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.643632 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.643702 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.646884 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:51.646887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:51.646889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:51.646911 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:51.648625 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.649171 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.649177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.649181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.649267 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.649604 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.649609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.649611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.649798 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:51.649904 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:51.650362 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 04:39:51.650481 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 04:39:51.651391 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.638856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36407 (local address 127.0.98.2:39349)
0924 04:39:51.639258 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:51.639296 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:51.640423 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:51.640893 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:51.640899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:51.641308 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:51.641748 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.641760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.642485 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.642487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.643319 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.643328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.644088 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.644094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.644239 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.644843 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:51.644866 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:51.646725 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:51.648775 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:51.648779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:51.648802 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:51.649040 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:51.649372 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:51.649380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:51.649381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:51.649499 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:51.649912 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:51.649920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:51.650138 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 04:39:51.650316 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 04:39:51.650615 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260924 04:39:51.652597 19920 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36407
I20260924 04:39:51.652815 19920 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:51.653420 19920 heartbeater.cc:511] Master 127.0.98.62:36407 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:39:51.654469 19595 ts_manager.cc:194] Registered new tserver with Master: 8225129f8b50490581a5cae3deb6e49d (127.0.98.2:45897)
I20260924 04:39:51.655025 19595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:39:51.671713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36407
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:51.779604 19925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:51.779827 19925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:51.779866 19925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:51.783676 19925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:51.783720 19925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:51.783772 19925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:39:51.788677 19925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:51.789528 19925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:51.790457 19925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:51.799311 19930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:51.799374 19931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:51.799311 19933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:51.799621 19925 server_base.cc:1034] running on GCE node
I20260924 04:39:51.800107 19925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:51.800683 19925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:51.801856 19925 hybrid_clock.cc:648] HybridClock initialized: now 1790224791801837 us; error 59 us; skew 500 ppm
Sep 24 04:39:51 dist-test-slave-jlnt krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:51.804600 19925 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:39:51.805451 19925 webserver.cc:460] Webserver started at http://127.0.98.3:37399/ using document root <none> and password file <none>
I20260924 04:39:51.805966 19925 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:51.806051 19925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:51.806255 19925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:51.808120 19925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "51381dc544d74a5fadbe26c5e02e6e14"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "22c8283943f781569d276d1e78771990"
server_key_iv: "dcc3479d4afeb574cb33174f8e3ebc0f"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.808656 19925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51381dc544d74a5fadbe26c5e02e6e14"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "22c8283943f781569d276d1e78771990"
server_key_iv: "dcc3479d4afeb574cb33174f8e3ebc0f"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.812069 19925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:39:51.814497 19940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.815639 19925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:39:51.815764 19925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "51381dc544d74a5fadbe26c5e02e6e14"
format_stamp: "Formatted at 2026-09-24 04:39:51 on dist-test-slave-jlnt"
server_key: "22c8283943f781569d276d1e78771990"
server_key_iv: "dcc3479d4afeb574cb33174f8e3ebc0f"
server_key_version: "encryptionkey@0"
I20260924 04:39:51.815894 19925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:51.827142 19925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:51.829700 19925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:51.829846 19925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:51.830291 19925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:51.830983 19925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:51.831027 19925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.831063 19925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:51.831086 19925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:51.839962 19925 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42889
I20260924 04:39:51.840006 20053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42889 every 8 connection(s)
I20260924 04:39:51.840762 19925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:39:51.847337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 19925
I20260924 04:39:51.847466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:39:51.847781 392 external_mini_cluster.cc:1428] Setting key 08e2021369ddab7cb70d4734525d33ba
Sep 24 04:39:51 dist-test-slave-jlnt krb5kdc[19550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:39:51.853595 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.842409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52193 (local address 127.0.98.62:36407)
0924 04:39:51.842547 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:51.842551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:51.843273 (+ 722us) server_negotiation.cc:407] Connection header received
0924 04:39:51.844436 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:51.844439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:51.844484 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:51.844555 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:51.845660 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.846196 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.847280 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.847627 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.847746 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.850564 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:51.850568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:51.850569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:51.850592 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:51.851820 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.852269 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.852272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.852273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.852344 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.852659 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.852661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.852662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.852818 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:51.852891 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:51.853389 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 04:39:51.853485 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:39:51.854346 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.842725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36407 (local address 127.0.98.3:52193)
0924 04:39:51.843125 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:51.843165 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:51.844225 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:51.844707 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:51.844713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:51.845030 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:51.845509 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.845522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.846304 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.846307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.847142 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.847150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.847770 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.847772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.847953 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.848566 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:39:51.848590 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:51.850422 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:51.851936 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:51.851941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:51.851950 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:51.852184 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:51.852443 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:39:51.852446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:39:51.852447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:39:51.852576 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:39:51.852904 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:51.852908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:51.853124 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 04:39:51.853342 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 04:39:51.853568 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260924 04:39:51.855430 20054 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36407
I20260924 04:39:51.855660 20054 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:51.856112 20054 heartbeater.cc:511] Master 127.0.98.62:36407 requested a full tablet report, sending...
I20260924 04:39:51.857048 19595 ts_manager.cc:194] Registered new tserver with Master: 51381dc544d74a5fadbe26c5e02e6e14 (127.0.98.3:42889)
I20260924 04:39:51.857606 19595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52193
I20260924 04:39:51.861653 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:39:51.870257 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.863502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.0.98.62:36407)
0924 04:39:51.863638 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:51.863643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:51.863840 (+ 197us) server_negotiation.cc:407] Connection header received
0924 04:39:51.863930 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:51.863933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:51.863986 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:51.864050 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:51.864673 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.865217 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.865979 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.866178 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.866253 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.867504 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:51.867507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:51.867509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:51.867525 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:51.868703 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.869061 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.869063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.869064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.869131 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:51.869432 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:51.869434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:51.869435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:51.869620 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:51.869718 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:51.869974 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 04:39:51.870058 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:39:51.872937 19595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:39:51.876058 19595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:39:51.889837 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.883513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35483 (local address 127.0.0.1:32964)
0924 04:39:51.883917 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:51.883965 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:51.884080 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:51.885466 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:51.885472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:51.885493 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:51.885618 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.885627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.886706 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.886710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.888118 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.888128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.889225 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.889229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.889354 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.889399 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 04:39:51.889540 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 04:39:51.889676 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":82,"threads_started":1}
I20260924 04:39:51.889854 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.884233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.0.98.1:35483)
0924 04:39:51.884946 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:51.884957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:51.884984 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:39:51.885059 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:51.885065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:51.885312 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:51.885475 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:51.885748 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.886574 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.888261 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.889064 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.889215 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.889368 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:51.889571 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 04:39:51.889665 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":133,"threads_started":1}
I20260924 04:39:51.890339 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.884023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45897 (local address 127.0.0.1:48306)
0924 04:39:51.885057 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:51.885069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:51.885182 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:51.885644 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:51.885649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:51.885670 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:51.885859 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.885864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.887398 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.887401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.888503 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.888510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.889974 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.889977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.890065 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.890084 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:39:51.890164 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:39:51.890226 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":82,"thread_start_us":98,"threads_started":1}
I20260924 04:39:51.890609 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.884630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.0.98.3:42889)
0924 04:39:51.884989 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:51.885003 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:51.885026 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:39:51.885098 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:51.885101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:51.885354 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:51.885743 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:51.885895 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.887179 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.888760 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.889550 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.890195 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.890323 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:51.890373 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:39:51.890475 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":160,"threads_started":1}
I20260924 04:39:51.890955 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.884438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.0.98.2:45897)
0924 04:39:51.884757 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:51.884771 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:51.885171 (+ 400us) server_negotiation.cc:407] Connection header received
0924 04:39:51.885283 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:51.885288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:51.885467 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:51.885612 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:51.885978 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.887254 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.888619 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.889451 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.889568 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.889719 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:51.890221 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 04:39:51.890792 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
I20260924 04:39:51.890098 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:51.883530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42889 (local address 127.0.0.1:44102)
0924 04:39:51.884335 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:51.884350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:51.884444 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:51.885549 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:51.885554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:51.885578 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:51.885715 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.885721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.887362 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.887368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.888629 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:51.888640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:51.889675 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:51.889679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:51.889780 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:51.889834 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 04:39:51.889887 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:39:51.889955 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":20,"thread_start_us":93,"threads_started":1}
I20260924 04:39:51.893122 19720 tablet_service.cc:1467] Processing CreateTablet for tablet 73e0be568ce643db91029c9cc1134519 (DEFAULT_TABLE table=test-table [id=2da55859968a43028d171aaad0005bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:51.893551 19988 tablet_service.cc:1467] Processing CreateTablet for tablet 73e0be568ce643db91029c9cc1134519 (DEFAULT_TABLE table=test-table [id=2da55859968a43028d171aaad0005bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:51.893663 19854 tablet_service.cc:1467] Processing CreateTablet for tablet 73e0be568ce643db91029c9cc1134519 (DEFAULT_TABLE table=test-table [id=2da55859968a43028d171aaad0005bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:51.894268 19720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e0be568ce643db91029c9cc1134519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:51.894532 19854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e0be568ce643db91029c9cc1134519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:51.894532 19988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e0be568ce643db91029c9cc1134519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:51.900753 20071 tablet_bootstrap.cc:492] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a: Bootstrap starting.
I20260924 04:39:51.901958 20073 tablet_bootstrap.cc:492] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: Bootstrap starting.
I20260924 04:39:51.903264 20071 tablet_bootstrap.cc:654] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:51.903738 20073 tablet_bootstrap.cc:654] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:51.904462 20071 log.cc:824] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:51.904839 20073 log.cc:824] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:51.905618 20072 tablet_bootstrap.cc:492] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: Bootstrap starting.
I20260924 04:39:51.906737 20073 tablet_bootstrap.cc:492] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: No bootstrap required, opened a new log
I20260924 04:39:51.906737 20071 tablet_bootstrap.cc:492] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a: No bootstrap required, opened a new log
I20260924 04:39:51.906921 20073 ts_tablet_manager.cc:1397] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 04:39:51.906921 20071 ts_tablet_manager.cc:1397] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:39:51.907449 20072 tablet_bootstrap.cc:654] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:51.908453 20072 log.cc:824] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:51.910322 20072 tablet_bootstrap.cc:492] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: No bootstrap required, opened a new log
I20260924 04:39:51.910538 20072 ts_tablet_manager.cc:1397] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:39:51.910789 20071 raft_consensus.cc:348] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } }
I20260924 04:39:51.910789 20073 raft_consensus.cc:348] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } }
I20260924 04:39:51.911051 20073 raft_consensus.cc:374] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:51.911051 20071 raft_consensus.cc:374] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:51.911113 20071 raft_consensus.cc:729] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e1eb7f1dc34e05a36db6f43d602b1a, State: Initialized, Role: FOLLOWER
I20260924 04:39:51.911139 20073 raft_consensus.cc:729] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51381dc544d74a5fadbe26c5e02e6e14, State: Initialized, Role: FOLLOWER
I20260924 04:39:51.911597 20073 consensus_queue.cc:260] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } }
I20260924 04:39:51.911669 20071 consensus_queue.cc:260] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } }
I20260924 04:39:51.912303 20054 heartbeater.cc:503] Master 127.0.98.62:36407 was elected leader, sending a full tablet report...
I20260924 04:39:51.912552 19786 heartbeater.cc:503] Master 127.0.98.62:36407 was elected leader, sending a full tablet report...
I20260924 04:39:51.912609 20071 ts_tablet_manager.cc:1428] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:39:51.912608 20073 ts_tablet_manager.cc:1428] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 04:39:51.915377 20072 raft_consensus.cc:348] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } }
I20260924 04:39:51.915611 20072 raft_consensus.cc:374] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:51.915670 20072 raft_consensus.cc:729] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8225129f8b50490581a5cae3deb6e49d, State: Initialized, Role: FOLLOWER
I20260924 04:39:51.916144 20072 consensus_queue.cc:260] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } }
I20260924 04:39:51.917169 20072 ts_tablet_manager.cc:1428] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260924 04:39:51.917291 19920 heartbeater.cc:503] Master 127.0.98.62:36407 was elected leader, sending a full tablet report...
W20260924 04:39:51.933758 19787 tablet.cc:2369] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:39:52.091635 20055 tablet.cc:2369] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:39:52.137924 19921 tablet.cc:2369] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:39:52.260730 20077 raft_consensus.cc:482] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:52.261001 20077 raft_consensus.cc:504] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } }
I20260924 04:39:52.262290 20077 leader_election.cc:288] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e1eb7f1dc34e05a36db6f43d602b1a (127.0.98.1:35483), 8225129f8b50490581a5cae3deb6e49d (127.0.98.2:45897)
I20260924 04:39:52.267454 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:52.262676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38675 (local address 127.0.98.2:45897)
0924 04:39:52.262818 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:52.262822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:52.263054 (+ 232us) server_negotiation.cc:407] Connection header received
0924 04:39:52.263117 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:52.263122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:52.263171 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:52.263242 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:52.263535 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.264308 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.266198 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.266807 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.266929 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:52.266997 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:52.267258 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 04:39:52.267326 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:39:52.267454 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:52.262585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45397 (local address 127.0.98.1:35483)
0924 04:39:52.262753 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:52.262759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:52.262775 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:39:52.262981 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:52.262984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:52.263060 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:52.263148 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:52.263522 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.264323 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.266198 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.266807 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.266957 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:52.267009 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:52.267258 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 04:39:52.267329 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 04:39:52.267467 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:52.262491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35483 (local address 127.0.98.3:45397)
0924 04:39:52.262701 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:52.262716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:52.262813 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:52.263179 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:52.263181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:52.263194 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:52.263364 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:52.263373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.264430 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.264433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:52.266060 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:52.266071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.266965 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.266968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:52.267112 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:52.267156 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 04:39:52.267223 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:39:52.267307 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":268}
I20260924 04:39:52.267473 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:52.262582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45897 (local address 127.0.98.3:38675)
0924 04:39:52.262954 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:52.262964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:52.263053 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:52.263265 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:52.263267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:52.263278 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:52.263424 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:52.263432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.264430 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.264433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:52.266060 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:52.266071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.266940 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.266945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:52.267098 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:52.267143 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 04:39:52.267223 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:39:52.267307 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":173,"thread_start_us":128,"threads_started":1}
I20260924 04:39:52.268247 19740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e0be568ce643db91029c9cc1134519" candidate_uuid: "51381dc544d74a5fadbe26c5e02e6e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" is_pre_election: true
I20260924 04:39:52.268296 19874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e0be568ce643db91029c9cc1134519" candidate_uuid: "51381dc544d74a5fadbe26c5e02e6e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8225129f8b50490581a5cae3deb6e49d" is_pre_election: true
I20260924 04:39:52.268571 19874 raft_consensus.cc:2454] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51381dc544d74a5fadbe26c5e02e6e14 in term 0.
I20260924 04:39:52.268571 19740 raft_consensus.cc:2454] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51381dc544d74a5fadbe26c5e02e6e14 in term 0.
I20260924 04:39:52.269039 19941 leader_election.cc:302] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51381dc544d74a5fadbe26c5e02e6e14, e5e1eb7f1dc34e05a36db6f43d602b1a; no voters:
I20260924 04:39:52.269284 20077 raft_consensus.cc:2781] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:39:52.269379 20077 raft_consensus.cc:482] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:39:52.269439 20077 raft_consensus.cc:3037] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:52.270445 20077 raft_consensus.cc:504] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } }
I20260924 04:39:52.270805 20077 leader_election.cc:288] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [CANDIDATE]: Term 1 election: Requested vote from peers e5e1eb7f1dc34e05a36db6f43d602b1a (127.0.98.1:35483), 8225129f8b50490581a5cae3deb6e49d (127.0.98.2:45897)
I20260924 04:39:52.271176 19740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e0be568ce643db91029c9cc1134519" candidate_uuid: "51381dc544d74a5fadbe26c5e02e6e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a"
I20260924 04:39:52.271198 19874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e0be568ce643db91029c9cc1134519" candidate_uuid: "51381dc544d74a5fadbe26c5e02e6e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8225129f8b50490581a5cae3deb6e49d"
I20260924 04:39:52.271303 19874 raft_consensus.cc:3037] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:52.271303 19740 raft_consensus.cc:3037] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:52.272536 19740 raft_consensus.cc:2454] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51381dc544d74a5fadbe26c5e02e6e14 in term 1.
I20260924 04:39:52.272536 19874 raft_consensus.cc:2454] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51381dc544d74a5fadbe26c5e02e6e14 in term 1.
I20260924 04:39:52.272825 19944 leader_election.cc:302] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51381dc544d74a5fadbe26c5e02e6e14, 8225129f8b50490581a5cae3deb6e49d, e5e1eb7f1dc34e05a36db6f43d602b1a; no voters:
I20260924 04:39:52.273020 20077 raft_consensus.cc:2781] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:52.273300 20077 raft_consensus.cc:686] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 1 LEADER]: Becoming Leader. State: Replica: 51381dc544d74a5fadbe26c5e02e6e14, State: Running, Role: LEADER
I20260924 04:39:52.273614 20077 consensus_queue.cc:237] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } }
I20260924 04:39:52.276894 19594 catalog_manager.cc:5495] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51381dc544d74a5fadbe26c5e02e6e14 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "51381dc544d74a5fadbe26c5e02e6e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51381dc544d74a5fadbe26c5e02e6e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 } health_report { overall_health: UNKNOWN } } }
I20260924 04:39:52.339718 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:52.336035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.0.98.3:42889)
0924 04:39:52.336181 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:52.336188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:52.336206 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:39:52.336410 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:52.336412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:52.336467 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:52.336583 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:39:52.336823 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.337401 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.338256 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.338476 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.338589 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:52.338900 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:39:52.339313 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:39:52.339448 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:52.339499 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 04:39:52.339554 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:39:52.345561 19874 raft_consensus.cc:1261] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [term 1 FOLLOWER]: Refusing update from remote peer 51381dc544d74a5fadbe26c5e02e6e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:52.345561 19740 raft_consensus.cc:1261] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [term 1 FOLLOWER]: Refusing update from remote peer 51381dc544d74a5fadbe26c5e02e6e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:52.346268 20081 consensus_queue.cc:1035] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e1eb7f1dc34e05a36db6f43d602b1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:52.346467 20077 consensus_queue.cc:1035] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8225129f8b50490581a5cae3deb6e49d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:52.354322 20086 mvcc.cc:205] Tried to move back new op lower bound from 7332760749439848448 to 7332760749155233792. Current Snapshot: MvccSnapshot[applied={T|T < 7332760749439848448}]
I20260924 04:39:52.361955 19593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:52.362128 19593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:52.364516 19593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d03ab1939bfc468a9d5473531228e6a3: Sending DeleteTablet for 3 replicas of tablet 73e0be568ce643db91029c9cc1134519
I20260924 04:39:52.365252 19720 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e0be568ce643db91029c9cc1134519 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32964
I20260924 04:39:52.365320 19854 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e0be568ce643db91029c9cc1134519 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48306
I20260924 04:39:52.365306 19988 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e0be568ce643db91029c9cc1134519 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44102
I20260924 04:39:52.365741 20093 tablet_replica.cc:330] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a: stopping tablet replica
I20260924 04:39:52.365759 20094 tablet_replica.cc:330] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: stopping tablet replica
I20260924 04:39:52.366045 20093 raft_consensus.cc:2229] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:52.366071 20094 raft_consensus.cc:2229] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:52.366377 20093 raft_consensus.cc:2258] T 73e0be568ce643db91029c9cc1134519 P e5e1eb7f1dc34e05a36db6f43d602b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:52.366389 20094 raft_consensus.cc:2258] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:52.366698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 19657
I20260924 04:39:52.366859 20095 tablet_replica.cc:330] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: stopping tablet replica
I20260924 04:39:52.367105 20095 raft_consensus.cc:2229] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:39:52.368161 20094 ts_tablet_manager.cc:1905] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:52.371026 20094 ts_tablet_manager.cc:1918] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:52.371122 20094 log.cc:1196] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-1/wal/wals/73e0be568ce643db91029c9cc1134519
I20260924 04:39:52.371424 20094 ts_tablet_manager.cc:1939] T 73e0be568ce643db91029c9cc1134519 P 8225129f8b50490581a5cae3deb6e49d: Deleting consensus metadata
I20260924 04:39:52.372603 19582 catalog_manager.cc:4841] TS 8225129f8b50490581a5cae3deb6e49d (127.0.98.2:45897): tablet 73e0be568ce643db91029c9cc1134519 (table test-table [id=2da55859968a43028d171aaad0005bde]) successfully deleted
I20260924 04:39:52.372787 20095 raft_consensus.cc:2258] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:52.374069 20095 ts_tablet_manager.cc:1905] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:52.376947 20095 ts_tablet_manager.cc:1918] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:52.377079 20095 log.cc:1196] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/73e0be568ce643db91029c9cc1134519
I20260924 04:39:52.377434 20095 ts_tablet_manager.cc:1939] T 73e0be568ce643db91029c9cc1134519 P 51381dc544d74a5fadbe26c5e02e6e14: Deleting consensus metadata
I20260924 04:39:52.378572 19580 catalog_manager.cc:4841] TS 51381dc544d74a5fadbe26c5e02e6e14 (127.0.98.3:42889): tablet 73e0be568ce643db91029c9cc1134519 (table test-table [id=2da55859968a43028d171aaad0005bde]) successfully deleted
W20260924 04:39:52.380338 19579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35483 (error 108)
I20260924 04:39:52.381275 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:52.380961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35483 (local address 127.0.0.1:32978)
0924 04:39:52.381097 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:52.381191 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:39:52.381443 19579 catalog_manager.cc:4584] TS e5e1eb7f1dc34e05a36db6f43d602b1a (127.0.98.1:35483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73e0be568ce643db91029c9cc1134519: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35483: connect: Connection refused (error 111)
I20260924 04:39:52.381505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 19791
I20260924 04:39:52.388932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 19925
I20260924 04:39:52.394304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 19563
2026-09-24T04:39:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2576 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T04:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:52Z Disabled control of system clock
I20260924 04:39:52.423542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38277
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:39:52.531725 20100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:52.531978 20100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:52.532012 20100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:52.535996 20100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:52.536047 20100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:52.536062 20100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:52.536118 20100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:52.536145 20100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:52.540807 20100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:52.541832 20100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:52.543133 20100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:52.551343 20105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:52.551450 20100 server_base.cc:1034] running on GCE node
W20260924 04:39:52.551362 20108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:52.551600 20106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:52.552135 20100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:52.553262 20100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:52.554445 20100 hybrid_clock.cc:648] HybridClock initialized: now 1790224792554429 us; error 31 us; skew 500 ppm
I20260924 04:39:52.556180 20100 webserver.cc:460] Webserver started at http://127.0.98.62:36193/ using document root <none> and password file <none>
I20260924 04:39:52.556730 20100 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:52.556819 20100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:52.557052 20100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:52.559053 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "94a2e71393444fa68aa422597f90357e"
format_stamp: "Formatted at 2026-09-24 04:39:52 on dist-test-slave-jlnt"
server_key: "f990d6fef712aac5ac5e74b64d92646d"
server_key_iv: "30e6a53df21a877820cb2032433e7547"
server_key_version: "encryptionkey@0"
I20260924 04:39:52.559597 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "94a2e71393444fa68aa422597f90357e"
format_stamp: "Formatted at 2026-09-24 04:39:52 on dist-test-slave-jlnt"
server_key: "f990d6fef712aac5ac5e74b64d92646d"
server_key_iv: "30e6a53df21a877820cb2032433e7547"
server_key_version: "encryptionkey@0"
I20260924 04:39:52.563068 20100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:39:52.565362 20114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:52.566527 20100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:52.566663 20100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "94a2e71393444fa68aa422597f90357e"
format_stamp: "Formatted at 2026-09-24 04:39:52 on dist-test-slave-jlnt"
server_key: "f990d6fef712aac5ac5e74b64d92646d"
server_key_iv: "30e6a53df21a877820cb2032433e7547"
server_key_version: "encryptionkey@0"
I20260924 04:39:52.566771 20100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:52.583577 20100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:52.584203 20100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:52.584391 20100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:52.590806 20100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38277
I20260924 04:39:52.590816 20166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38277 every 8 connection(s)
I20260924 04:39:52.591693 20100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:52.594627 20167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:52.598450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 20100
I20260924 04:39:52.598572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:52.598848 392 external_mini_cluster.cc:1428] Setting key d3bafcd4dd3880ef86745e9c67b84e47
I20260924 04:39:52.600244 20167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e: Bootstrap starting.
I20260924 04:39:52.602398 20167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:52.603112 20167 log.cc:824] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:52.604789 20167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e: No bootstrap required, opened a new log
I20260924 04:39:52.607784 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:52.600255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.0.98.62:38277)
0924 04:39:52.600868 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:52.600881 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:52.600915 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:39:52.601586 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:52.601604 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:52.601980 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:52.602306 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:52.602731 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.603749 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.604626 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.605179 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.605357 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:52.606204 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:52.606209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:52.606221 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:52.606577 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:52.606627 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:52.606645 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:52.606754 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:52.606895 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:39:52.607132 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260924 04:39:52.608417 20167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a2e71393444fa68aa422597f90357e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } }
I20260924 04:39:52.608639 20167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:52.608702 20167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a2e71393444fa68aa422597f90357e, State: Initialized, Role: FOLLOWER
I20260924 04:39:52.609119 20167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a2e71393444fa68aa422597f90357e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } }
I20260924 04:39:52.609290 20167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:52.609369 20167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:52.609478 20167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:52.610404 20167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a2e71393444fa68aa422597f90357e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } }
I20260924 04:39:52.610721 20167 leader_election.cc:302] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94a2e71393444fa68aa422597f90357e; no voters:
I20260924 04:39:52.610996 20167 leader_election.cc:288] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:52.611079 20172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:52.611336 20172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [term 1 LEADER]: Becoming Leader. State: Replica: 94a2e71393444fa68aa422597f90357e, State: Running, Role: LEADER
I20260924 04:39:52.611675 20172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a2e71393444fa68aa422597f90357e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } }
I20260924 04:39:52.612129 20167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:52.613336 20174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94a2e71393444fa68aa422597f90357e. Latest consensus state: current_term: 1 leader_uuid: "94a2e71393444fa68aa422597f90357e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a2e71393444fa68aa422597f90357e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } }
I20260924 04:39:52.613479 20174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:52.613765 20173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94a2e71393444fa68aa422597f90357e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a2e71393444fa68aa422597f90357e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } }
I20260924 04:39:52.613921 20173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e [sys.catalog]: This master's current role is: LEADER
W20260924 04:39:52.616571 20187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:39:52.616657 20187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:39:52.616734 20181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:52.617501 20181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:52.622195 20181 catalog_manager.cc:1347] Generated new cluster ID: 7b6c064403ae4a6b9a02b3c219f63c0c
I20260924 04:39:52.622256 20181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:52.638833 20181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:52.639652 20181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:52.645535 20181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e: Generated new TSK 0
I20260924 04:39:52.646193 20181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:39:52.654723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38277
--builtin_ntp_servers=127.0.98.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:39:52.763234 20191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:52.763458 20191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:52.763500 20191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:52.767457 20191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:52.767511 20191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:52.767562 20191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:39:52.772202 20191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:52.773231 20191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:52.774550 20191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:52.783948 20199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:52.783991 20197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:52.783988 20196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:52.784199 20191 server_base.cc:1034] running on GCE node
I20260924 04:39:52.784644 20191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:52.785357 20191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:52.786553 20191 hybrid_clock.cc:648] HybridClock initialized: now 1790224792786519 us; error 46 us; skew 500 ppm
I20260924 04:39:52.788317 20191 webserver.cc:460] Webserver started at http://127.0.98.1:35429/ using document root <none> and password file <none>
I20260924 04:39:52.788870 20191 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:52.788961 20191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:52.789242 20191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:52.791306 20191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/data/instance:
uuid: "a88787e25f70448eab14dcf4083ab347"
format_stamp: "Formatted at 2026-09-24 04:39:52 on dist-test-slave-jlnt"
server_key: "9ce4842a6afb6ad2553bd192563b3704"
server_key_iv: "4173fd95e6af77b8dff2e319f64a0fc2"
server_key_version: "encryptionkey@0"
I20260924 04:39:52.791860 20191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a88787e25f70448eab14dcf4083ab347"
format_stamp: "Formatted at 2026-09-24 04:39:52 on dist-test-slave-jlnt"
server_key: "9ce4842a6afb6ad2553bd192563b3704"
server_key_iv: "4173fd95e6af77b8dff2e319f64a0fc2"
server_key_version: "encryptionkey@0"
I20260924 04:39:52.795408 20191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:39:52.797698 20205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:52.798832 20191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:52.798964 20191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/wal
uuid: "a88787e25f70448eab14dcf4083ab347"
format_stamp: "Formatted at 2026-09-24 04:39:52 on dist-test-slave-jlnt"
server_key: "9ce4842a6afb6ad2553bd192563b3704"
server_key_iv: "4173fd95e6af77b8dff2e319f64a0fc2"
server_key_version: "encryptionkey@0"
I20260924 04:39:52.799069 20191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:52.818254 20191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:52.818881 20191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:52.819068 20191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:52.819623 20191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:52.820551 20191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:52.820617 20191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:52.820678 20191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:52.820729 20191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:52.829289 20191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42523
I20260924 04:39:52.829319 20318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42523 every 8 connection(s)
I20260924 04:39:52.830124 20191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:39:52.830732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 20191
I20260924 04:39:52.830849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:39:52.831115 392 external_mini_cluster.cc:1428] Setting key b6ceae0040d140f87f11fbb87c111d2e
I20260924 04:39:52.833164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38277
--builtin_ntp_servers=127.0.98.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:39:52.841257 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:52.831913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57597 (local address 127.0.98.62:38277)
0924 04:39:52.832084 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:52.832089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:52.833340 (+ 1251us) server_negotiation.cc:407] Connection header received
0924 04:39:52.835252 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:52.835256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:52.835306 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:52.835388 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:52.836545 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.837189 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.839157 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.839368 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.839456 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:52.840749 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:52.840752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:52.840754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:52.840766 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:52.840783 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:52.840790 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:52.840846 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:52.841046 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 04:39:52.841092 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 04:39:52.842433 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:52.832428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38277 (local address 127.0.98.1:57597)
0924 04:39:52.833193 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:52.833249 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:52.835052 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:52.835564 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:52.835575 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:52.835704 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:52.836377 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:52.836399 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.837343 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.837350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:52.838946 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:52.838962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:52.839533 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:52.839539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:52.839775 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:52.840436 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:39:52.840440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:52.840498 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:39:52.840588 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:52.840883 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:52.840895 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:52.840903 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 04:39:52.841044 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 04:39:52.841376 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260924 04:39:52.843698 20319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38277
I20260924 04:39:52.843930 20319 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:52.844411 20319 heartbeater.cc:511] Master 127.0.98.62:38277 requested a full tablet report, sending...
I20260924 04:39:52.845901 20131 ts_manager.cc:194] Registered new tserver with Master: a88787e25f70448eab14dcf4083ab347 (127.0.98.1:42523)
I20260924 04:39:52.847097 20131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57597
W20260924 04:39:52.942111 20322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:52.942350 20322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:52.942387 20322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:52.946382 20322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:52.946429 20322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:52.946511 20322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:39:52.951120 20322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:52.952064 20322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:52.953421 20322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:52.962752 20330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:52.962764 20327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:52.962747 20328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:52.963158 20322 server_base.cc:1034] running on GCE node
I20260924 04:39:52.963479 20322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:52.964088 20322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:52.965206 20322 hybrid_clock.cc:648] HybridClock initialized: now 1790224792965189 us; error 32 us; skew 500 ppm
I20260924 04:39:52.966864 20322 webserver.cc:460] Webserver started at http://127.0.98.2:41883/ using document root <none> and password file <none>
I20260924 04:39:52.967407 20322 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:52.967468 20322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:52.967711 20322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:52.969703 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/data/instance:
uuid: "004354d9919c4446a1e4bc2dc391326f"
format_stamp: "Formatted at 2026-09-24 04:39:52 on dist-test-slave-jlnt"
server_key: "345db69eebfed10e15a5d8d7179883b0"
server_key_iv: "c6e11d5645f18caee5dbeb086f37a5b7"
server_key_version: "encryptionkey@0"
I20260924 04:39:52.970315 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/wal/instance:
uuid: "004354d9919c4446a1e4bc2dc391326f"
format_stamp: "Formatted at 2026-09-24 04:39:52 on dist-test-slave-jlnt"
server_key: "345db69eebfed10e15a5d8d7179883b0"
server_key_iv: "c6e11d5645f18caee5dbeb086f37a5b7"
server_key_version: "encryptionkey@0"
I20260924 04:39:52.973805 20322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:39:52.976037 20336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:52.977295 20322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 04:39:52.977424 20322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/wal
uuid: "004354d9919c4446a1e4bc2dc391326f"
format_stamp: "Formatted at 2026-09-24 04:39:52 on dist-test-slave-jlnt"
server_key: "345db69eebfed10e15a5d8d7179883b0"
server_key_iv: "c6e11d5645f18caee5dbeb086f37a5b7"
server_key_version: "encryptionkey@0"
I20260924 04:39:52.977530 20322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:52.991842 20322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:52.992404 20322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:52.992542 20322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:52.993013 20322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:52.993907 20322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:52.993952 20322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:52.993996 20322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:52.994019 20322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:53.002629 20322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38765
I20260924 04:39:53.002660 20449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38765 every 8 connection(s)
I20260924 04:39:53.003562 20322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:39:53.009507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 20322
I20260924 04:39:53.009630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:39:53.009948 392 external_mini_cluster.cc:1428] Setting key 1e779cb4c1d4fb243f8ff2fd3db2a99a
I20260924 04:39:53.012058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38277
--builtin_ntp_servers=127.0.98.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:39:53.012948 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.005306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43883 (local address 127.0.98.62:38277)
0924 04:39:53.005455 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:53.005459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:53.006459 (+ 1000us) server_negotiation.cc:407] Connection header received
0924 04:39:53.007732 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:53.007735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:53.007779 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:53.007851 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:53.008652 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.009224 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.010801 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.011159 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.011267 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.012485 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:53.012488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:53.012490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:53.012502 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:53.012521 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:53.012529 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:53.012589 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:53.012768 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 04:39:53.012814 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:39:53.014088 20452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.005713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38277 (local address 127.0.98.2:43883)
0924 04:39:53.006305 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:53.006345 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:53.007556 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:53.008005 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:53.008012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:53.008088 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:53.008537 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.008548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.009348 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.009356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.010681 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.010698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.011302 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.011306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.011500 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.012151 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:39:53.012155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:53.012210 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:39:53.012288 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:53.012616 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:53.012627 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:53.012637 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 04:39:53.012764 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 04:39:53.013187 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":105,"threads_started":1}
I20260924 04:39:53.015154 20450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38277
I20260924 04:39:53.015372 20450 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:53.015873 20450 heartbeater.cc:511] Master 127.0.98.62:38277 requested a full tablet report, sending...
I20260924 04:39:53.016865 20131 ts_manager.cc:194] Registered new tserver with Master: 004354d9919c4446a1e4bc2dc391326f (127.0.98.2:38765)
I20260924 04:39:53.017480 20131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43883
W20260924 04:39:53.119483 20453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:53.119748 20453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:53.119824 20453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:53.123780 20453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:53.123857 20453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:53.123942 20453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:39:53.128594 20453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:53.129583 20453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:53.130896 20453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:53.139763 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:53.139776 20458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:53.139804 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:53.140132 20453 server_base.cc:1034] running on GCE node
I20260924 04:39:53.140573 20453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:53.141129 20453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:53.142339 20453 hybrid_clock.cc:648] HybridClock initialized: now 1790224793142325 us; error 24 us; skew 500 ppm
I20260924 04:39:53.143992 20453 webserver.cc:460] Webserver started at http://127.0.98.3:43937/ using document root <none> and password file <none>
I20260924 04:39:53.144552 20453 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:53.144613 20453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:53.144846 20453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:53.146792 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/data/instance:
uuid: "0cbcb10c07364e56a3682dc1864ea477"
format_stamp: "Formatted at 2026-09-24 04:39:53 on dist-test-slave-jlnt"
server_key: "81989d67a69bf0dbae73ca99f5b63496"
server_key_iv: "745710408b01523c41eaef159b928bfe"
server_key_version: "encryptionkey@0"
I20260924 04:39:53.147322 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0cbcb10c07364e56a3682dc1864ea477"
format_stamp: "Formatted at 2026-09-24 04:39:53 on dist-test-slave-jlnt"
server_key: "81989d67a69bf0dbae73ca99f5b63496"
server_key_iv: "745710408b01523c41eaef159b928bfe"
server_key_version: "encryptionkey@0"
I20260924 04:39:53.150781 20453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:39:53.153015 20467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:53.154222 20453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:53.154361 20453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/wal
uuid: "0cbcb10c07364e56a3682dc1864ea477"
format_stamp: "Formatted at 2026-09-24 04:39:53 on dist-test-slave-jlnt"
server_key: "81989d67a69bf0dbae73ca99f5b63496"
server_key_iv: "745710408b01523c41eaef159b928bfe"
server_key_version: "encryptionkey@0"
I20260924 04:39:53.154459 20453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:53.180092 20453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:53.180783 20453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:53.180974 20453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:53.181607 20453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:39:53.182485 20453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:39:53.182530 20453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:53.182593 20453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:39:53.182626 20453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:53.191160 20453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44651
I20260924 04:39:53.191186 20580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44651 every 8 connection(s)
I20260924 04:39:53.192029 20453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:39:53.198632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 20453
I20260924 04:39:53.198724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:39:53.199015 392 external_mini_cluster.cc:1428] Setting key abb2b74d8cb1daf18459e0b3df9c1ebc
I20260924 04:39:53.209481 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.193734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46331 (local address 127.0.98.62:38277)
0924 04:39:53.193867 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:53.193870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:53.194791 (+ 921us) server_negotiation.cc:407] Connection header received
0924 04:39:53.196168 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:53.196172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:53.196232 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:53.196335 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:53.197209 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.197735 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.198911 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.199204 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.199317 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.208922 (+ 9605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:53.208926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:53.208928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:53.208940 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:53.208961 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:53.208969 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:53.209050 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:53.209270 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 04:39:53.209341 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:39:53.210400 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.194047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38277 (local address 127.0.98.3:46331)
0924 04:39:53.194640 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:53.194677 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:53.195953 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:53.196505 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:53.196511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:53.196590 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:39:53.197038 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.197049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.197860 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.197864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.198786 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.198797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.199351 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.199355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.199524 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.208541 (+ 9017us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:39:53.208543 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:39:53.208587 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:39:53.208657 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:39:53.209065 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:39:53.209078 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:39:53.209084 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:39:53.209270 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 04:39:53.209584 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260924 04:39:53.211561 20581 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38277
I20260924 04:39:53.211833 20581 heartbeater.cc:464] Registering TS with master...
I20260924 04:39:53.212523 20581 heartbeater.cc:511] Master 127.0.98.62:38277 requested a full tablet report, sending...
I20260924 04:39:53.213546 20131 ts_manager.cc:194] Registered new tserver with Master: 0cbcb10c07364e56a3682dc1864ea477 (127.0.98.3:44651)
I20260924 04:39:53.214161 20131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46331
I20260924 04:39:53.224232 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:39:53.229506 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.226126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.0.98.62:38277)
0924 04:39:53.226273 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:53.226277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:53.226416 (+ 139us) server_negotiation.cc:407] Connection header received
0924 04:39:53.226497 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:53.226499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:53.226535 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:53.226591 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:53.226805 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.227283 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.228062 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.228259 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.228334 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.229087 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:53.229090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:53.229092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:53.229101 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:53.229114 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:53.229120 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:53.229186 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:53.229285 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 04:39:53.229327 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:39:53.231997 20131 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:39:53.234906 20131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:39:53.248358 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.243287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.0.98.1:42523)
0924 04:39:53.244095 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:53.244103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:53.244122 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:39:53.244183 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:53.244186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:53.244318 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:53.244495 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:53.244880 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.245693 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.247004 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.247707 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.247831 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.248038 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:53.248122 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:39:53.248214 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":90,"threads_started":1}
I20260924 04:39:53.248347 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.243131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42523 (local address 127.0.0.1:46478)
0924 04:39:53.243847 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:53.243861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:53.243925 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:53.244643 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:53.244646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:53.244663 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:53.244762 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.244767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.245817 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.245820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.246866 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.246875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.247831 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.247835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.247944 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.247980 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:39:53.248093 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 04:39:53.248205 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":85,"threads_started":1}
I20260924 04:39:53.248766 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.242596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44651 (local address 127.0.0.1:57938)
0924 04:39:53.243467 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:53.243515 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:53.243628 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:53.244566 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:53.244569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:53.244587 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:53.244709 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.244719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.246188 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.246191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.247289 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.247298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.248340 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.248344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.248480 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.248501 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:39:53.248563 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:39:53.248625 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":53,"thread_start_us":73,"threads_started":1}
I20260924 04:39:53.248979 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.242681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38765 (local address 127.0.0.1:44682)
0924 04:39:53.243658 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:53.243672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:53.243739 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:53.244476 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:53.244482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:53.244505 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:53.244634 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.244641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.246066 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.246069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.247345 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.247353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.248340 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.248344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.248678 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.248705 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:39:53.248771 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:39:53.248848 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":233,"thread_start_us":106,"threads_started":1}
I20260924 04:39:53.249403 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.243615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.0.98.2:38765)
0924 04:39:53.243991 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:53.244005 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:53.244030 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:39:53.244109 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:53.244115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:53.244311 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:53.244502 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:53.244760 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.245934 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.247435 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.248200 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.248500 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.248674 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:53.249185 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 04:39:53.249284 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":158,"threads_started":1}
I20260924 04:39:53.249495 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.242691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.0.98.3:44651)
0924 04:39:53.244188 (+ 1497us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:53.244198 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:53.244216 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:39:53.244273 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:53.244277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:53.244424 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:53.244857 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:53.244989 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.246079 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.247432 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.248200 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.249015 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.249190 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:53.249236 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:39:53.249316 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1409,"thread_start_us":89,"threads_started":1}
I20260924 04:39:53.251827 20384 tablet_service.cc:1467] Processing CreateTablet for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff (DEFAULT_TABLE table=test-table [id=e52c4ae4f2564060a25aa83e1aec5e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:53.251680 20253 tablet_service.cc:1467] Processing CreateTablet for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff (DEFAULT_TABLE table=test-table [id=e52c4ae4f2564060a25aa83e1aec5e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:53.251992 20515 tablet_service.cc:1467] Processing CreateTablet for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff (DEFAULT_TABLE table=test-table [id=e52c4ae4f2564060a25aa83e1aec5e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:39:53.252715 20384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:53.252738 20253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:53.252777 20515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:53.259276 20598 tablet_bootstrap.cc:492] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f: Bootstrap starting.
I20260924 04:39:53.260134 20600 tablet_bootstrap.cc:492] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: Bootstrap starting.
I20260924 04:39:53.260216 20599 tablet_bootstrap.cc:492] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347: Bootstrap starting.
I20260924 04:39:53.261751 20598 tablet_bootstrap.cc:654] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:53.261878 20600 tablet_bootstrap.cc:654] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:53.262184 20599 tablet_bootstrap.cc:654] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:53.262578 20600 log.cc:824] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:53.262585 20598 log.cc:824] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:53.262894 20599 log.cc:824] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:53.264418 20599 tablet_bootstrap.cc:492] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347: No bootstrap required, opened a new log
I20260924 04:39:53.264446 20600 tablet_bootstrap.cc:492] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: No bootstrap required, opened a new log
I20260924 04:39:53.264451 20598 tablet_bootstrap.cc:492] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f: No bootstrap required, opened a new log
I20260924 04:39:53.264622 20600 ts_tablet_manager.cc:1397] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:39:53.264622 20598 ts_tablet_manager.cc:1397] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:39:53.264652 20599 ts_tablet_manager.cc:1397] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:39:53.267730 20599 raft_consensus.cc:348] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } }
I20260924 04:39:53.267956 20599 raft_consensus.cc:374] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:53.268029 20599 raft_consensus.cc:729] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88787e25f70448eab14dcf4083ab347, State: Initialized, Role: FOLLOWER
I20260924 04:39:53.268484 20599 consensus_queue.cc:260] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } }
I20260924 04:39:53.268754 20600 raft_consensus.cc:348] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } }
I20260924 04:39:53.268754 20598 raft_consensus.cc:348] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } }
I20260924 04:39:53.268988 20598 raft_consensus.cc:374] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:53.269070 20598 raft_consensus.cc:729] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004354d9919c4446a1e4bc2dc391326f, State: Initialized, Role: FOLLOWER
I20260924 04:39:53.269083 20319 heartbeater.cc:503] Master 127.0.98.62:38277 was elected leader, sending a full tablet report...
I20260924 04:39:53.269182 20600 raft_consensus.cc:374] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:53.269233 20600 raft_consensus.cc:729] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cbcb10c07364e56a3682dc1864ea477, State: Initialized, Role: FOLLOWER
I20260924 04:39:53.269454 20599 ts_tablet_manager.cc:1428] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:39:53.269505 20598 consensus_queue.cc:260] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } }
I20260924 04:39:53.269632 20600 consensus_queue.cc:260] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } }
I20260924 04:39:53.270176 20450 heartbeater.cc:503] Master 127.0.98.62:38277 was elected leader, sending a full tablet report...
I20260924 04:39:53.270519 20598 ts_tablet_manager.cc:1428] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 04:39:53.270509 20600 ts_tablet_manager.cc:1428] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260924 04:39:53.270792 20581 heartbeater.cc:503] Master 127.0.98.62:38277 was elected leader, sending a full tablet report...
I20260924 04:39:53.290808 20606 raft_consensus.cc:482] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:39:53.291002 20606 raft_consensus.cc:504] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } }
I20260924 04:39:53.291991 20606 leader_election.cc:288] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 004354d9919c4446a1e4bc2dc391326f (127.0.98.2:38765), a88787e25f70448eab14dcf4083ab347 (127.0.98.1:42523)
I20260924 04:39:53.296797 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.292235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52521 (local address 127.0.98.2:38765)
0924 04:39:53.292357 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:53.292362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:53.292439 (+ 77us) server_negotiation.cc:407] Connection header received
0924 04:39:53.292542 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:53.292545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:53.292614 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:53.292691 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:53.292996 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.293800 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.295557 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.296146 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.296255 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.296310 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:53.296614 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 04:39:53.296680 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:39:53.296797 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.292283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44173 (local address 127.0.98.1:42523)
0924 04:39:53.292393 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:53.292399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:53.292619 (+ 220us) server_negotiation.cc:407] Connection header received
0924 04:39:53.292721 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:53.292724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:53.292771 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:53.292840 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:39:53.293125 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.293851 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.295557 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.296150 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.296255 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.296310 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:53.296614 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 04:39:53.296680 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:39:53.296828 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.292174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38765 (local address 127.0.98.3:52521)
0924 04:39:53.292341 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:53.292356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:53.292446 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:53.292725 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:53.292727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:53.292741 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:53.292878 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.292885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.293924 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.293927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.295429 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.295440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.296267 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.296270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.296441 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.296508 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 04:39:53.296586 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:39:53.296682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":266}
I20260924 04:39:53.296844 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.292199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42523 (local address 127.0.98.3:44173)
0924 04:39:53.292528 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:53.292539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:39:53.292621 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:39:53.292878 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:39:53.292881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:39:53.292894 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:39:53.293022 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.293028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.293944 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.293947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.295429 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:39:53.295452 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.296267 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.296270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:39:53.296424 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.296519 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 04:39:53.296586 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:39:53.296682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":190,"thread_start_us":104,"threads_started":1}
I20260924 04:39:53.297420 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f18524cb86d4bf99d7028ac5a4dc4ff" candidate_uuid: "0cbcb10c07364e56a3682dc1864ea477" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004354d9919c4446a1e4bc2dc391326f" is_pre_election: true
I20260924 04:39:53.297685 20404 raft_consensus.cc:2454] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cbcb10c07364e56a3682dc1864ea477 in term 0.
I20260924 04:39:53.298151 20470 leader_election.cc:302] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004354d9919c4446a1e4bc2dc391326f, 0cbcb10c07364e56a3682dc1864ea477; no voters:
I20260924 04:39:53.298365 20273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f18524cb86d4bf99d7028ac5a4dc4ff" candidate_uuid: "0cbcb10c07364e56a3682dc1864ea477" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88787e25f70448eab14dcf4083ab347" is_pre_election: true
I20260924 04:39:53.298408 20606 raft_consensus.cc:2781] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:39:53.298502 20606 raft_consensus.cc:482] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:39:53.298565 20606 raft_consensus.cc:3037] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:53.298641 20273 raft_consensus.cc:2454] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cbcb10c07364e56a3682dc1864ea477 in term 0.
I20260924 04:39:53.299623 20606 raft_consensus.cc:504] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } }
I20260924 04:39:53.299948 20606 leader_election.cc:288] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [CANDIDATE]: Term 1 election: Requested vote from peers 004354d9919c4446a1e4bc2dc391326f (127.0.98.2:38765), a88787e25f70448eab14dcf4083ab347 (127.0.98.1:42523)
I20260924 04:39:53.300289 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f18524cb86d4bf99d7028ac5a4dc4ff" candidate_uuid: "0cbcb10c07364e56a3682dc1864ea477" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004354d9919c4446a1e4bc2dc391326f"
I20260924 04:39:53.300387 20404 raft_consensus.cc:3037] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:53.301282 20273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f18524cb86d4bf99d7028ac5a4dc4ff" candidate_uuid: "0cbcb10c07364e56a3682dc1864ea477" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88787e25f70448eab14dcf4083ab347"
I20260924 04:39:53.301380 20273 raft_consensus.cc:3037] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:53.301556 20404 raft_consensus.cc:2454] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cbcb10c07364e56a3682dc1864ea477 in term 1.
I20260924 04:39:53.301865 20470 leader_election.cc:302] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004354d9919c4446a1e4bc2dc391326f, 0cbcb10c07364e56a3682dc1864ea477; no voters:
I20260924 04:39:53.302021 20606 raft_consensus.cc:2781] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:53.302217 20606 raft_consensus.cc:686] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 1 LEADER]: Becoming Leader. State: Replica: 0cbcb10c07364e56a3682dc1864ea477, State: Running, Role: LEADER
I20260924 04:39:53.302609 20606 consensus_queue.cc:237] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } }
I20260924 04:39:53.302695 20273 raft_consensus.cc:2454] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cbcb10c07364e56a3682dc1864ea477 in term 1.
I20260924 04:39:53.305598 20130 catalog_manager.cc:5495] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cbcb10c07364e56a3682dc1864ea477 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0cbcb10c07364e56a3682dc1864ea477" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cbcb10c07364e56a3682dc1864ea477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 } health_report { overall_health: UNKNOWN } } }
I20260924 04:39:53.328962 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.324130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.0.98.3:44651)
0924 04:39:53.324235 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:53.324240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:53.324261 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:39:53.324487 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:53.324492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:53.324555 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:53.324670 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:39:53.324963 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.325842 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.327240 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:53.327614 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:53.327724 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:53.328116 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:39:53.328559 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:39:53.328648 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:53.328770 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 04:39:53.328814 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 04:39:53.331437 20320 tablet.cc:2369] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:39:53.339707 20404 raft_consensus.cc:1261] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [term 1 FOLLOWER]: Refusing update from remote peer 0cbcb10c07364e56a3682dc1864ea477: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:53.339846 20273 raft_consensus.cc:1261] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347 [term 1 FOLLOWER]: Refusing update from remote peer 0cbcb10c07364e56a3682dc1864ea477: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:39:53.340466 20608 consensus_queue.cc:1035] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [LEADER]: Connected to new peer: Peer: permanent_uuid: "004354d9919c4446a1e4bc2dc391326f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:53.340715 20606 consensus_queue.cc:1035] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88787e25f70448eab14dcf4083ab347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:39:53.348497 20613 mvcc.cc:205] Tried to move back new op lower bound from 7332760753499930624 to 7332760753370415104. Current Snapshot: MvccSnapshot[applied={T|T < 7332760753499930624}]
I20260924 04:39:53.349566 20612 mvcc.cc:205] Tried to move back new op lower bound from 7332760753499930624 to 7332760753370415104. Current Snapshot: MvccSnapshot[applied={T|T < 7332760753499930624}]
I20260924 04:39:53.355808 20130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:53.355962 20130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:39:53.358208 20130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94a2e71393444fa68aa422597f90357e: Sending DeleteTablet for 3 replicas of tablet 0f18524cb86d4bf99d7028ac5a4dc4ff
I20260924 04:39:53.358908 20515 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:53 UTC) from {username='slave'} at 127.0.0.1:57938
I20260924 04:39:53.359069 20384 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:53 UTC) from {username='slave'} at 127.0.0.1:44682
I20260924 04:39:53.359123 20253 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:39:53 UTC) from {username='slave'} at 127.0.0.1:46478
I20260924 04:39:53.359400 20622 tablet_replica.cc:330] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: stopping tablet replica
I20260924 04:39:53.359543 20624 tablet_replica.cc:330] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347: stopping tablet replica
I20260924 04:39:53.359758 20624 raft_consensus.cc:2229] T 0f18524cb86d4bf99d7028ac5a4dc4ff P a88787e25f70448eab14dcf4083ab347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:53.359745 20622 raft_consensus.cc:2229] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:39:53.359897 20623 tablet_replica.cc:330] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f: stopping tablet replica
I20260924 04:39:53.360200 20623 raft_consensus.cc:2229] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:39:53.360181 20622 raft_consensus.cc:2258] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:53.360251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 20191
I20260924 04:39:53.363235 20623 raft_consensus.cc:2258] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:39:53.363767 20622 ts_tablet_manager.cc:1905] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:53.367179 20623 ts_tablet_manager.cc:1905] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 004354d9919c4446a1e4bc2dc391326f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:39:53.367180 20622 ts_tablet_manager.cc:1918] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:39:53.367365 20622 log.cc:1196] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: Deleting WAL directory at /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790224545298817-392-0/minicluster-data/ts-2/wal/wals/0f18524cb86d4bf99d7028ac5a4dc4ff
I20260924 04:39:53.367759 20622 ts_tablet_manager.cc:1939] T 0f18524cb86d4bf99d7028ac5a4dc4ff P 0cbcb10c07364e56a3682dc1864ea477: Deleting consensus metadata
W20260924 04:39:53.368355 20116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42523 (error 108)
I20260924 04:39:53.369045 20115 catalog_manager.cc:4841] TS 0cbcb10c07364e56a3682dc1864ea477 (127.0.98.3:44651): tablet 0f18524cb86d4bf99d7028ac5a4dc4ff (table test-table [id=e52c4ae4f2564060a25aa83e1aec5e01]) successfully deleted
I20260924 04:39:53.369113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 20322
I20260924 04:39:53.374130 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.373836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42523 (local address 127.0.0.1:46494)
0924 04:39:53.373975 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:53.374045 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 04:39:53.374358 20116 catalog_manager.cc:4584] TS a88787e25f70448eab14dcf4083ab347 (127.0.98.1:42523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f18524cb86d4bf99d7028ac5a4dc4ff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42523: connect: Connection refused (error 111)
I20260924 04:39:53.375278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 20453
I20260924 04:39:53.376384 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:53.375495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38765 (local address 127.0.0.1:44690)
0924 04:39:53.376266 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0924 04:39:53.376319 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":690}
I20260924 04:39:53.382107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 20100
2026-09-24T04:39:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (987 ms)
[----------] 6 tests from EncryptionPolicyTest (10293 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:39:53 dist-test-slave-jlnt krb5kdc[20628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:39:53 dist-test-slave-jlnt krb5kdc[20628](info): set up 2 sockets
Sep 24 04:39:53 dist-test-slave-jlnt krb5kdc[20628](info): commencing operation
krb5kdc: starting...
W20260924 04:39:54.413216 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:39:54 dist-test-slave-jlnt krb5kdc[20628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:39:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:39:54.531689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:54.638525 20641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:54.638746 20641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:54.638785 20641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:54.642699 20641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:54.642745 20641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:54.642763 20641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:54.642779 20641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:54.642828 20641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:54.647855 20641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:54.648763 20641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:54.649755 20641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:54.657778 20647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:54.657860 20646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:54.657848 20641 server_base.cc:1034] running on GCE node
W20260924 04:39:54.657757 20649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:54.658667 20641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:54.659667 20641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:54.660853 20641 hybrid_clock.cc:648] HybridClock initialized: now 1790224794660838 us; error 36 us; skew 500 ppm
Sep 24 04:39:54 dist-test-slave-jlnt krb5kdc[20628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:54.663460 20641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:39:54.664258 20641 webserver.cc:460] Webserver started at http://127.0.0.1:32897/ using document root <none> and password file <none>
I20260924 04:39:54.664777 20641 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:54.664867 20641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:54.665096 20641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:54.667083 20641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "af65f9940370436b9a95f653c6d142b0"
format_stamp: "Formatted at 2026-09-24 04:39:54 on dist-test-slave-jlnt"
server_key: "d95c5548f2e2cc0f472c3bd64e0edf8b"
server_key_iv: "c905b012d7243cc94cfdce54c7197a25"
server_key_version: "encryptionkey@0"
I20260924 04:39:54.667624 20641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "af65f9940370436b9a95f653c6d142b0"
format_stamp: "Formatted at 2026-09-24 04:39:54 on dist-test-slave-jlnt"
server_key: "d95c5548f2e2cc0f472c3bd64e0edf8b"
server_key_iv: "c905b012d7243cc94cfdce54c7197a25"
server_key_version: "encryptionkey@0"
I20260924 04:39:54.671059 20641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:39:54.673272 20656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:54.674393 20641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:54.674531 20641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "af65f9940370436b9a95f653c6d142b0"
format_stamp: "Formatted at 2026-09-24 04:39:54 on dist-test-slave-jlnt"
server_key: "d95c5548f2e2cc0f472c3bd64e0edf8b"
server_key_iv: "c905b012d7243cc94cfdce54c7197a25"
server_key_version: "encryptionkey@0"
I20260924 04:39:54.674641 20641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:54.690243 20641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:54.692746 20641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:54.692941 20641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:54.699592 20641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40409
I20260924 04:39:54.699604 20708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40409 every 8 connection(s)
I20260924 04:39:54.700465 20641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:54.703034 20709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:54.707103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 20641
I20260924 04:39:54.707250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:54.707528 392 external_mini_cluster.cc:1428] Setting key f3767f62d8c8e6256d0611fc6424f5a1
I20260924 04:39:54.708411 20709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0: Bootstrap starting.
I20260924 04:39:54.710539 20709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:54.711431 20709 log.cc:824] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:54.713351 20709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0: No bootstrap required, opened a new log
Sep 24 04:39:54 dist-test-slave-jlnt krb5kdc[20628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:39:54.717468 20709 raft_consensus.cc:348] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af65f9940370436b9a95f653c6d142b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40409 } }
I20260924 04:39:54.717684 20709 raft_consensus.cc:374] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:54.717761 20709 raft_consensus.cc:729] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af65f9940370436b9a95f653c6d142b0, State: Initialized, Role: FOLLOWER
I20260924 04:39:54.718252 20709 consensus_queue.cc:260] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af65f9940370436b9a95f653c6d142b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40409 } }
I20260924 04:39:54.718406 20709 raft_consensus.cc:388] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:54.718497 20709 raft_consensus.cc:482] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:54.718593 20709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:54.719358 20709 raft_consensus.cc:504] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af65f9940370436b9a95f653c6d142b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40409 } }
I20260924 04:39:54.719671 20709 leader_election.cc:302] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af65f9940370436b9a95f653c6d142b0; no voters:
I20260924 04:39:54.719973 20709 leader_election.cc:288] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:54.720052 20714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:54.720249 20714 raft_consensus.cc:686] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [term 1 LEADER]: Becoming Leader. State: Replica: af65f9940370436b9a95f653c6d142b0, State: Running, Role: LEADER
I20260924 04:39:54.720578 20714 consensus_queue.cc:237] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af65f9940370436b9a95f653c6d142b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40409 } }
I20260924 04:39:54.721179 20709 sys_catalog.cc:564] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:54.721748 20716 sys_catalog.cc:455] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af65f9940370436b9a95f653c6d142b0. Latest consensus state: current_term: 1 leader_uuid: "af65f9940370436b9a95f653c6d142b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af65f9940370436b9a95f653c6d142b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40409 } } }
I20260924 04:39:54.721880 20716 sys_catalog.cc:458] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:54.722414 20718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:54.722503 20715 sys_catalog.cc:455] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af65f9940370436b9a95f653c6d142b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af65f9940370436b9a95f653c6d142b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40409 } } }
I20260924 04:39:54.722653 20715 sys_catalog.cc:458] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:54.722711 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:54.708854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48661 (local address 127.0.0.1:40409)
0924 04:39:54.709253 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:54.709268 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:54.709300 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:39:54.709905 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:54.709921 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:54.710225 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:54.710489 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:54.711381 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:54.712587 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:54.713710 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:54.714152 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:54.714265 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:54.716962 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:54.716967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:54.716989 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:54.717014 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:54.718973 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:54.719462 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:54.719468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:54.719483 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:54.719612 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:54.719955 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:54.719959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:54.719960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:54.720515 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:54.720724 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:54.721326 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 04:39:54.721506 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":114,"threads_started":1}
I20260924 04:39:54.725489 20718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:54.725709 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:39:54.732672 20718 catalog_manager.cc:1347] Generated new cluster ID: d002f74b4f354e02aec16c541bef59bb
I20260924 04:39:54.732728 20718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:54.739300 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:54.727870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58153 (local address 127.0.0.1:40409)
0924 04:39:54.727972 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:54.727987 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:54.731088 (+ 3101us) server_negotiation.cc:407] Connection header received
0924 04:39:54.731261 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:54.731266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:54.731314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:54.731383 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:54.732047 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:54.732726 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:54.733919 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:54.734287 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:54.734378 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:54.735986 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:54.735990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:54.735992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:54.736016 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:54.737622 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:54.738129 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:54.738132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:54.738133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:54.738206 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:54.738526 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:54.738529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:54.738530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:54.738696 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:54.738815 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:54.739080 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 04:39:54.739162 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:39:54.743135 20718 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:54.744210 20718 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:54.755290 20718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af65f9940370436b9a95f653c6d142b0: Generated new TSK 0
I20260924 04:39:54.756052 20718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:39:54.761060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 20641
W20260924 04:39:54.765599 20626 connection.cc:535] client connection to 127.0.0.1:40409 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40409): Connection reset by peer (error 104)
2026-09-24T04:39:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:39:54 dist-test-slave-jlnt krb5kdc[20740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:39:54 dist-test-slave-jlnt krb5kdc[20740](info): set up 2 sockets
Sep 24 04:39:54 dist-test-slave-jlnt krb5kdc[20740](info): commencing operation
krb5kdc: starting...
W20260924 04:39:55.805269 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:39:55 dist-test-slave-jlnt krb5kdc[20740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:39:55.922350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:56.030328 20753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:56.030565 20753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:56.030604 20753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:56.034557 20753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:56.034612 20753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:56.034631 20753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:56.034648 20753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:56.034663 20753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:56.039628 20753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:56.040500 20753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:56.041451 20753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:39:56.049419 20761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:56.049620 20753 server_base.cc:1034] running on GCE node
W20260924 04:39:56.049396 20759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:56.049396 20758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:56.050356 20753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:56.051294 20753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:56.052516 20753 hybrid_clock.cc:648] HybridClock initialized: now 1790224796052485 us; error 43 us; skew 500 ppm
Sep 24 04:39:56 dist-test-slave-jlnt krb5kdc[20740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:56.055167 20753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:39:56.056025 20753 webserver.cc:460] Webserver started at http://127.0.0.1:42485/ using document root <none> and password file <none>
I20260924 04:39:56.056561 20753 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:56.056653 20753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:56.056876 20753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:56.058859 20753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "87d0aad845b845789a88ee2e0d334ef7"
format_stamp: "Formatted at 2026-09-24 04:39:56 on dist-test-slave-jlnt"
server_key: "be655de12ca210e8e7a58ad053533eff"
server_key_iv: "ec6e1553d5f0b3e1d01eef7ed8178007"
server_key_version: "encryptionkey@0"
I20260924 04:39:56.059419 20753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "87d0aad845b845789a88ee2e0d334ef7"
format_stamp: "Formatted at 2026-09-24 04:39:56 on dist-test-slave-jlnt"
server_key: "be655de12ca210e8e7a58ad053533eff"
server_key_iv: "ec6e1553d5f0b3e1d01eef7ed8178007"
server_key_version: "encryptionkey@0"
I20260924 04:39:56.062943 20753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 04:39:56.065204 20768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:56.066330 20753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:56.066486 20753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "87d0aad845b845789a88ee2e0d334ef7"
format_stamp: "Formatted at 2026-09-24 04:39:56 on dist-test-slave-jlnt"
server_key: "be655de12ca210e8e7a58ad053533eff"
server_key_iv: "ec6e1553d5f0b3e1d01eef7ed8178007"
server_key_version: "encryptionkey@0"
I20260924 04:39:56.066599 20753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:56.082984 20753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:56.085834 20753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:56.086031 20753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:56.092968 20753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34947
I20260924 04:39:56.093004 20820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34947 every 8 connection(s)
I20260924 04:39:56.093915 20753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:56.096400 20821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:56.097711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 20753
I20260924 04:39:56.097857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:56.098150 392 external_mini_cluster.cc:1428] Setting key 944f77cb06883ac2cd8fa0fa797914d5
I20260924 04:39:56.102257 20821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7: Bootstrap starting.
I20260924 04:39:56.104334 20821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:56.105034 20821 log.cc:824] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:56.106802 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:56.099469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44789 (local address 127.0.0.1:34947)
0924 04:39:56.099909 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:56.099923 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:56.099957 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:39:56.100564 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:56.100579 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:56.100890 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:56.101248 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:56.101641 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:56.102624 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:56.103494 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:56.103947 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:56.104136 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:56.105021 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:56.105026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:56.105037 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:56.105546 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:56.105611 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:56.105645 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:56.105777 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:56.105954 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 04:39:56.106127 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":136,"threads_started":1}
I20260924 04:39:56.107139 20821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7: No bootstrap required, opened a new log
I20260924 04:39:56.109558 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:39:56.110909 20821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d0aad845b845789a88ee2e0d334ef7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34947 } }
I20260924 04:39:56.111194 20821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:56.111261 20821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d0aad845b845789a88ee2e0d334ef7, State: Initialized, Role: FOLLOWER
I20260924 04:39:56.111681 20821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d0aad845b845789a88ee2e0d334ef7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34947 } }
I20260924 04:39:56.111896 20821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:56.111999 20821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:56.112120 20821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:56.113191 20821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d0aad845b845789a88ee2e0d334ef7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34947 } }
I20260924 04:39:56.113517 20821 leader_election.cc:302] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d0aad845b845789a88ee2e0d334ef7; no voters:
I20260924 04:39:56.113783 20821 leader_election.cc:288] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:56.113883 20831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:56.114069 20831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 87d0aad845b845789a88ee2e0d334ef7, State: Running, Role: LEADER
I20260924 04:39:56.114418 20831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d0aad845b845789a88ee2e0d334ef7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34947 } }
I20260924 04:39:56.114980 20821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:56.115626 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d0aad845b845789a88ee2e0d334ef7. Latest consensus state: current_term: 1 leader_uuid: "87d0aad845b845789a88ee2e0d334ef7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d0aad845b845789a88ee2e0d334ef7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34947 } } }
I20260924 04:39:56.115749 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:56.116391 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:56.111768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59953 (local address 127.0.0.1:34947)
0924 04:39:56.111874 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:56.111878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:56.112028 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:39:56.112146 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:56.112150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:56.112196 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:56.112285 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:56.112493 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:56.113249 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:56.114342 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:56.114668 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:56.114773 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:56.115642 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:56.115648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:56.115650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:56.115664 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:56.115707 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:56.115723 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:56.115822 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:56.116222 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 04:39:56.116267 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:39:56.116686 20840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:56.116353 20832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87d0aad845b845789a88ee2e0d334ef7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d0aad845b845789a88ee2e0d334ef7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34947 } } }
I20260924 04:39:56.117384 20832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:56.119675 20840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:56.124965 20840 catalog_manager.cc:1347] Generated new cluster ID: 01ac8680d0a34860ab0b7dd0955d4070
I20260924 04:39:56.125046 20840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:56.135425 20840 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:56.136270 20840 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:56.148790 20840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87d0aad845b845789a88ee2e0d334ef7: Generated new TSK 0
I20260924 04:39:56.149408 20840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:39:56.157779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 20753
W20260924 04:39:56.163103 20738 connection.cc:535] client connection to 127.0.0.1:34947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34947): Connection reset by peer (error 104)
2026-09-24T04:39:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:39:56 dist-test-slave-jlnt krb5kdc[20852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:39:56 dist-test-slave-jlnt krb5kdc[20852](info): set up 2 sockets
Sep 24 04:39:56 dist-test-slave-jlnt krb5kdc[20852](info): commencing operation
krb5kdc: starting...
W20260924 04:39:57.186127 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:39:57 dist-test-slave-jlnt krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:39:57.307914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:57.414572 20865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:57.414829 20865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:57.414923 20865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:57.418905 20865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:57.418980 20865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:57.419024 20865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:57.419090 20865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:57.419130 20865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:57.424288 20865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:57.425319 20865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:57.426249 20865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:39:57.434231 20865 server_base.cc:1034] running on GCE node
W20260924 04:39:57.434437 20873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:57.434492 20871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:57.434492 20870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:57.435114 20865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:57.436103 20865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:57.437276 20865 hybrid_clock.cc:648] HybridClock initialized: now 1790224797437261 us; error 29 us; skew 500 ppm
Sep 24 04:39:57 dist-test-slave-jlnt krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:57.439924 20865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:39:57.440747 20865 webserver.cc:460] Webserver started at http://127.0.0.1:40033/ using document root <none> and password file <none>
I20260924 04:39:57.441334 20865 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:57.441424 20865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:57.441639 20865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:57.443495 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "09f3a329877740b6aeaf4554b0cd84fc"
format_stamp: "Formatted at 2026-09-24 04:39:57 on dist-test-slave-jlnt"
server_key: "911f89abfc06c3b09ba097767b5c53e8"
server_key_iv: "cb9cc3a97cb10647d5f592c43a111e05"
server_key_version: "encryptionkey@0"
I20260924 04:39:57.444032 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "09f3a329877740b6aeaf4554b0cd84fc"
format_stamp: "Formatted at 2026-09-24 04:39:57 on dist-test-slave-jlnt"
server_key: "911f89abfc06c3b09ba097767b5c53e8"
server_key_iv: "cb9cc3a97cb10647d5f592c43a111e05"
server_key_version: "encryptionkey@0"
I20260924 04:39:57.447430 20865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:39:57.449654 20880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:57.450759 20865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:39:57.450901 20865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "09f3a329877740b6aeaf4554b0cd84fc"
format_stamp: "Formatted at 2026-09-24 04:39:57 on dist-test-slave-jlnt"
server_key: "911f89abfc06c3b09ba097767b5c53e8"
server_key_iv: "cb9cc3a97cb10647d5f592c43a111e05"
server_key_version: "encryptionkey@0"
I20260924 04:39:57.451007 20865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:57.455262 20865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:57.457962 20865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:57.458145 20865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:57.464844 20865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34561
I20260924 04:39:57.464869 20932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34561 every 8 connection(s)
I20260924 04:39:57.465777 20865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:57.468297 20933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:57.472852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 20865
I20260924 04:39:57.473014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:57.473341 392 external_mini_cluster.cc:1428] Setting key bb35a381d62ce99ab18abd5c517679c2
I20260924 04:39:57.474095 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc: Bootstrap starting.
I20260924 04:39:57.477157 20933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:57.478102 20933 log.cc:824] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:39:57.479882 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:57.474887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40991 (local address 127.0.0.1:34561)
0924 04:39:57.475373 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:57.475397 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:57.475429 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:39:57.476009 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:57.476023 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:57.476282 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:57.477128 (+ 846us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:57.478276 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:57.478284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:57.478298 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:57.478309 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:57.478388 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:57.478413 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:57.478530 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:57.478698 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 04:39:57.478853 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260924 04:39:57.480357 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc: No bootstrap required, opened a new log
I20260924 04:39:57.482733 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:39:57.485212 20933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3a329877740b6aeaf4554b0cd84fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260924 04:39:57.485483 20933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:57.485555 20933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f3a329877740b6aeaf4554b0cd84fc, State: Initialized, Role: FOLLOWER
I20260924 04:39:57.485991 20933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3a329877740b6aeaf4554b0cd84fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260924 04:39:57.486136 20933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:57.486204 20933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:57.486320 20933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:57.486573 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:57.484963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47279 (local address 127.0.0.1:34561)
0924 04:39:57.485082 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:57.485087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:57.485268 (+ 181us) server_negotiation.cc:407] Connection header received
0924 04:39:57.485399 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:57.485401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:57.485425 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:57.485515 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:57.486192 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:57.486195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:57.486200 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:39:57.486214 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:57.486241 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:39:57.486252 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:57.486351 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:57.486451 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 04:39:57.486484 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:39:57.487355 20933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3a329877740b6aeaf4554b0cd84fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260924 04:39:57.487705 20933 leader_election.cc:302] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f3a329877740b6aeaf4554b0cd84fc; no voters:
I20260924 04:39:57.488018 20933 leader_election.cc:288] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:57.488144 20943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:57.488358 20943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [term 1 LEADER]: Becoming Leader. State: Replica: 09f3a329877740b6aeaf4554b0cd84fc, State: Running, Role: LEADER
I20260924 04:39:57.488619 20943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3a329877740b6aeaf4554b0cd84fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260924 04:39:57.489190 20933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:57.490113 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f3a329877740b6aeaf4554b0cd84fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3a329877740b6aeaf4554b0cd84fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } } }
I20260924 04:39:57.490204 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:57.490458 20945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f3a329877740b6aeaf4554b0cd84fc. Latest consensus state: current_term: 1 leader_uuid: "09f3a329877740b6aeaf4554b0cd84fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f3a329877740b6aeaf4554b0cd84fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } } }
I20260924 04:39:57.490543 20945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:57.490585 20952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:57.493544 20952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:57.498615 20952 catalog_manager.cc:1347] Generated new cluster ID: 76754008ea474814bd12f4e3b3606912
I20260924 04:39:57.498683 20952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:57.526837 20952 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:57.527190 20952 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:57.542897 20952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f3a329877740b6aeaf4554b0cd84fc: Generated new TSK 0
I20260924 04:39:57.543525 20952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:39:57.548514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 20865
2026-09-24T04:39:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:39:57 dist-test-slave-jlnt krb5kdc[20964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:39:57 dist-test-slave-jlnt krb5kdc[20964](info): set up 2 sockets
Sep 24 04:39:57 dist-test-slave-jlnt krb5kdc[20964](info): commencing operation
krb5kdc: starting...
W20260924 04:39:58.577479 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:39:58 dist-test-slave-jlnt krb5kdc[20964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:39:58.693922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:39:58.800752 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:39:58.800980 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:39:58.801048 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:39:58.804947 20977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:39:58.804998 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:39:58.805017 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:39:58.805033 20977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:39:58.805089 20977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:39:58.810119 20977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:39:58.811018 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:39:58.811939 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:39:58.820014 20977 server_base.cc:1034] running on GCE node
W20260924 04:39:58.819957 20982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:58.819975 20985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:39:58.819981 20983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:39:58.820767 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:39:58.821902 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:39:58.823071 20977 hybrid_clock.cc:648] HybridClock initialized: now 1790224798823051 us; error 31 us; skew 500 ppm
Sep 24 04:39:58 dist-test-slave-jlnt krb5kdc[20964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:39:58.837570 20977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:39:58.838384 20977 webserver.cc:460] Webserver started at http://127.0.0.1:43963/ using document root <none> and password file <none>
I20260924 04:39:58.838877 20977 fs_manager.cc:359] Metadata directory not provided
I20260924 04:39:58.838933 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:39:58.839087 20977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:39:58.840934 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "e05dc0f2407b4f63b9557aca12dcf1a9"
format_stamp: "Formatted at 2026-09-24 04:39:58 on dist-test-slave-jlnt"
server_key: "7bc145e8860009dc4b598a1e55a71f30"
server_key_iv: "ac95d299c9eabfc8a15d55f6e9a4346f"
server_key_version: "encryptionkey@0"
I20260924 04:39:58.841492 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "e05dc0f2407b4f63b9557aca12dcf1a9"
format_stamp: "Formatted at 2026-09-24 04:39:58 on dist-test-slave-jlnt"
server_key: "7bc145e8860009dc4b598a1e55a71f30"
server_key_iv: "ac95d299c9eabfc8a15d55f6e9a4346f"
server_key_version: "encryptionkey@0"
I20260924 04:39:58.844862 20977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:39:58.847072 20992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:39:58.848191 20977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:39:58.848315 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "e05dc0f2407b4f63b9557aca12dcf1a9"
format_stamp: "Formatted at 2026-09-24 04:39:58 on dist-test-slave-jlnt"
server_key: "7bc145e8860009dc4b598a1e55a71f30"
server_key_iv: "ac95d299c9eabfc8a15d55f6e9a4346f"
server_key_version: "encryptionkey@0"
I20260924 04:39:58.848414 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:39:58.875622 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:39:58.878250 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:39:58.878448 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:39:58.884852 20977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35111
I20260924 04:39:58.884923 21044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35111 every 8 connection(s)
I20260924 04:39:58.885763 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:39:58.888356 21045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:39:58.889078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 20977
I20260924 04:39:58.889293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:39:58.889640 392 external_mini_cluster.cc:1428] Setting key 51eb6fc2ac2a23f66173a0347f8d351a
I20260924 04:39:58.894667 21045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9: Bootstrap starting.
I20260924 04:39:58.896878 21045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9: Neither blocks nor log segments found. Creating new log.
I20260924 04:39:58.897620 21045 log.cc:824] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:39:58 dist-test-slave-jlnt krb5kdc[20964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:39:58.899503 21045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9: No bootstrap required, opened a new log
I20260924 04:39:58.902909 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:58.890964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41621 (local address 127.0.0.1:35111)
0924 04:39:58.891385 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:58.891401 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:58.891437 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:39:58.892093 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:58.892109 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:58.892458 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:58.892738 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:58.893749 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:58.894688 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:58.895554 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:58.896004 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:58.896170 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:58.898653 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:58.898658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:58.898670 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:58.898694 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:58.900275 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:58.900735 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:58.900740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:58.900744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:58.900843 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:58.901119 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:58.901124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:58.901126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:58.901530 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:58.901722 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:58.901997 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 04:39:58.902259 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":134,"threads_started":1}
I20260924 04:39:58.902921 21045 raft_consensus.cc:348] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05dc0f2407b4f63b9557aca12dcf1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } }
I20260924 04:39:58.903220 21045 raft_consensus.cc:374] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:39:58.903280 21045 raft_consensus.cc:729] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05dc0f2407b4f63b9557aca12dcf1a9, State: Initialized, Role: FOLLOWER
I20260924 04:39:58.903712 21045 consensus_queue.cc:260] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05dc0f2407b4f63b9557aca12dcf1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } }
I20260924 04:39:58.903885 21045 raft_consensus.cc:388] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:39:58.903952 21045 raft_consensus.cc:482] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:39:58.904040 21045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:39:58.905058 21045 raft_consensus.cc:504] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05dc0f2407b4f63b9557aca12dcf1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } }
I20260924 04:39:58.905411 21045 leader_election.cc:302] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e05dc0f2407b4f63b9557aca12dcf1a9; no voters:
I20260924 04:39:58.905728 21045 leader_election.cc:288] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:39:58.905813 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:39:58.905838 21050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:39:58.906054 21050 raft_consensus.cc:686] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [term 1 LEADER]: Becoming Leader. State: Replica: e05dc0f2407b4f63b9557aca12dcf1a9, State: Running, Role: LEADER
I20260924 04:39:58.906363 21050 consensus_queue.cc:237] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05dc0f2407b4f63b9557aca12dcf1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } }
I20260924 04:39:58.906996 21045 sys_catalog.cc:564] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:39:58.907752 21051 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e05dc0f2407b4f63b9557aca12dcf1a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05dc0f2407b4f63b9557aca12dcf1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } } }
I20260924 04:39:58.907876 21051 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:58.908215 21052 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e05dc0f2407b4f63b9557aca12dcf1a9. Latest consensus state: current_term: 1 leader_uuid: "e05dc0f2407b4f63b9557aca12dcf1a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05dc0f2407b4f63b9557aca12dcf1a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } } }
I20260924 04:39:58.908298 21052 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:39:58.908588 21063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:39:58.911623 21063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:39:58.917011 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:39:58.907918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40089 (local address 127.0.0.1:35111)
0924 04:39:58.908029 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:39:58.908034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:39:58.908447 (+ 413us) server_negotiation.cc:407] Connection header received
0924 04:39:58.908546 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:39:58.908552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:39:58.908603 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:39:58.908687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:39:58.909608 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:58.910362 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:58.911500 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:39:58.911857 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:39:58.911958 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:39:58.913537 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:39:58.913540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:39:58.913543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:39:58.913566 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:39:58.914986 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:58.915431 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:58.915434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:58.915436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:58.915526 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:39:58.916131 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:39:58.916134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:39:58.916136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:39:58.916336 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:39:58.916460 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:39:58.916704 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 04:39:58.916813 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:39:58.917440 21063 catalog_manager.cc:1347] Generated new cluster ID: 680055f0b05e496b93d8498f12ad3890
I20260924 04:39:58.917529 21063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:39:58.928200 21063 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:39:58.928995 21063 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:39:58.936803 21063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e05dc0f2407b4f63b9557aca12dcf1a9: Generated new TSK 0
I20260924 04:39:58.937439 21063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:39:58.942667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 20977
2026-09-24T04:39:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:39:58 dist-test-slave-jlnt krb5kdc[21076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:39:58 dist-test-slave-jlnt krb5kdc[21076](info): set up 2 sockets
Sep 24 04:39:58 dist-test-slave-jlnt krb5kdc[21076](info): commencing operation
krb5kdc: starting...
W20260924 04:39:59.977276 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:00 dist-test-slave-jlnt krb5kdc[21076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:40:00.095836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:00.200824 21089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:00.201094 21089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:00.201189 21089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:00.205042 21089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:00.205124 21089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:00.205180 21089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:00.205240 21089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:00.205281 21089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:00.210287 21089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:00.211215 21089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:00.212172 21089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:00.220074 21097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:00.220099 21094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:00.220292 21089 server_base.cc:1034] running on GCE node
W20260924 04:40:00.220114 21095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:00.220912 21089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:00.221983 21089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:00.223193 21089 hybrid_clock.cc:648] HybridClock initialized: now 1790224800223167 us; error 35 us; skew 500 ppm
Sep 24 04:40:00 dist-test-slave-jlnt krb5kdc[21076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:00.225924 21089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:40:00.226696 21089 webserver.cc:460] Webserver started at http://127.0.0.1:36063/ using document root <none> and password file <none>
I20260924 04:40:00.227209 21089 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:00.227293 21089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:00.227500 21089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:00.229434 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "ee2a1f855f3a47979c8d87793e4ed962"
format_stamp: "Formatted at 2026-09-24 04:40:00 on dist-test-slave-jlnt"
server_key: "ec7948744734ff5c53d4aba48fdf2d50"
server_key_iv: "446d0731ef3bd1594a5a5bdb176f7e25"
server_key_version: "encryptionkey@0"
I20260924 04:40:00.230005 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee2a1f855f3a47979c8d87793e4ed962"
format_stamp: "Formatted at 2026-09-24 04:40:00 on dist-test-slave-jlnt"
server_key: "ec7948744734ff5c53d4aba48fdf2d50"
server_key_iv: "446d0731ef3bd1594a5a5bdb176f7e25"
server_key_version: "encryptionkey@0"
I20260924 04:40:00.233358 21089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 04:40:00.235515 21104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:00.236630 21089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:40:00.236774 21089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "ee2a1f855f3a47979c8d87793e4ed962"
format_stamp: "Formatted at 2026-09-24 04:40:00 on dist-test-slave-jlnt"
server_key: "ec7948744734ff5c53d4aba48fdf2d50"
server_key_iv: "446d0731ef3bd1594a5a5bdb176f7e25"
server_key_version: "encryptionkey@0"
I20260924 04:40:00.236881 21089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:00.251875 21089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:00.254681 21089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:00.254870 21089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:00.261665 21089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39375
I20260924 04:40:00.261693 21156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39375 every 8 connection(s)
I20260924 04:40:00.262615 21089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:00.265127 21157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:00.270387 21157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962: Bootstrap starting.
I20260924 04:40:00.270726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 21089
I20260924 04:40:00.270869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:00.271143 392 external_mini_cluster.cc:1428] Setting key c653625e6d1ed57679fe818ea5f5077a
I20260924 04:40:00.272929 21157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:00.273698 21157 log.cc:824] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:00.275394 21157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962: No bootstrap required, opened a new log
I20260924 04:40:00.278967 21157 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2a1f855f3a47979c8d87793e4ed962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39375 } }
I20260924 04:40:00.279234 21157 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:00.279314 21157 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2a1f855f3a47979c8d87793e4ed962, State: Initialized, Role: FOLLOWER
I20260924 04:40:00.279727 21157 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2a1f855f3a47979c8d87793e4ed962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39375 } }
I20260924 04:40:00.279840 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:00.272413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50003 (local address 127.0.0.1:39375)
0924 04:40:00.272848 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:00.272862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:00.272895 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:40:00.273604 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:00.273622 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:00.273986 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:00.274295 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:00.274693 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:00.275703 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:00.276579 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:00.277032 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:00.277228 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:00.278119 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:00.278124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:00.278136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:00.278495 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:00.278543 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:00.278564 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:00.278673 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:00.278818 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 04:40:00.279006 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":159,"threads_started":1}
I20260924 04:40:00.279873 21157 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:00.280011 21157 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:00.280117 21157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:00.281005 21157 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2a1f855f3a47979c8d87793e4ed962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39375 } }
I20260924 04:40:00.281355 21157 leader_election.cc:302] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee2a1f855f3a47979c8d87793e4ed962; no voters:
I20260924 04:40:00.281692 21157 leader_election.cc:288] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:00.281785 21162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:00.282068 21162 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [term 1 LEADER]: Becoming Leader. State: Replica: ee2a1f855f3a47979c8d87793e4ed962, State: Running, Role: LEADER
I20260924 04:40:00.282418 21162 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2a1f855f3a47979c8d87793e4ed962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39375 } }
I20260924 04:40:00.282642 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:00.282920 21157 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:00.284664 21162 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee2a1f855f3a47979c8d87793e4ed962. Latest consensus state: current_term: 1 leader_uuid: "ee2a1f855f3a47979c8d87793e4ed962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2a1f855f3a47979c8d87793e4ed962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39375 } } }
I20260924 04:40:00.285079 21163 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee2a1f855f3a47979c8d87793e4ed962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2a1f855f3a47979c8d87793e4ed962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39375 } } }
I20260924 04:40:00.285645 21163 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:00.285521 21162 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962 [sys.catalog]: This master's current role is: LEADER
W20260924 04:40:00.287883 21180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:00.287961 21180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:40:00.288038 21182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:00.288805 21182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:00.289626 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:00.285205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38095 (local address 127.0.0.1:39375)
0924 04:40:00.285300 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:00.285304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:00.285463 (+ 159us) server_negotiation.cc:407] Connection header received
0924 04:40:00.285634 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:00.285637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:00.285679 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:00.285732 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:00.286122 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:00.286849 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:00.287941 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:00.288246 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:00.288355 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:00.289190 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:00.289193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:00.289196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:00.289208 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:00.289234 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:00.289247 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:00.289336 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:00.289454 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:40:00.289500 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:40:00.294057 21182 catalog_manager.cc:1347] Generated new cluster ID: 4366a341ba5a4e419aa4bfa94f69a89a
I20260924 04:40:00.294119 21182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:00.310971 21182 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:00.311781 21182 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:00.316800 21182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee2a1f855f3a47979c8d87793e4ed962: Generated new TSK 0
I20260924 04:40:00.317447 21182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:00.325637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 21089
2026-09-24T04:40:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:00 dist-test-slave-jlnt krb5kdc[21188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:00 dist-test-slave-jlnt krb5kdc[21188](info): set up 2 sockets
Sep 24 04:40:00 dist-test-slave-jlnt krb5kdc[21188](info): commencing operation
krb5kdc: starting...
W20260924 04:40:01.365322 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:01 dist-test-slave-jlnt krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:40:01.483588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:01.589097 21201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:01.589332 21201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:01.589371 21201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:01.593251 21201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:01.593299 21201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:01.593319 21201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:01.593336 21201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:01.593351 21201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:01.598444 21201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:01.599267 21201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:01.600142 21201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:01.608754 21209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:01.608752 21206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:01.608755 21207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:01.609198 21201 server_base.cc:1034] running on GCE node
I20260924 04:40:01.609750 21201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:01.610877 21201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:01.612115 21201 hybrid_clock.cc:648] HybridClock initialized: now 1790224801612103 us; error 30 us; skew 500 ppm
Sep 24 04:40:01 dist-test-slave-jlnt krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:01.614854 21201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:40:01.615672 21201 webserver.cc:460] Webserver started at http://127.0.0.1:33209/ using document root <none> and password file <none>
I20260924 04:40:01.616139 21201 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:01.616194 21201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:01.616349 21201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:01.618218 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "206359a69cf6414a820b45da3301c604"
format_stamp: "Formatted at 2026-09-24 04:40:01 on dist-test-slave-jlnt"
server_key: "e748ff37b5d7afe46da2163e1e5637d6"
server_key_iv: "7ec25f063766dbcdbb148a9968eb42ae"
server_key_version: "encryptionkey@0"
I20260924 04:40:01.618698 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "206359a69cf6414a820b45da3301c604"
format_stamp: "Formatted at 2026-09-24 04:40:01 on dist-test-slave-jlnt"
server_key: "e748ff37b5d7afe46da2163e1e5637d6"
server_key_iv: "7ec25f063766dbcdbb148a9968eb42ae"
server_key_version: "encryptionkey@0"
I20260924 04:40:01.622021 21201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:40:01.624228 21216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:01.625516 21201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:40:01.625639 21201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "206359a69cf6414a820b45da3301c604"
format_stamp: "Formatted at 2026-09-24 04:40:01 on dist-test-slave-jlnt"
server_key: "e748ff37b5d7afe46da2163e1e5637d6"
server_key_iv: "7ec25f063766dbcdbb148a9968eb42ae"
server_key_version: "encryptionkey@0"
I20260924 04:40:01.625737 21201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:01.630012 21201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:01.633023 21201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:01.633244 21201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:01.639801 21201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45807
I20260924 04:40:01.639840 21268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45807 every 8 connection(s)
I20260924 04:40:01.640625 21201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:01.643184 21269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:01.648367 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604: Bootstrap starting.
I20260924 04:40:01.648725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 21201
I20260924 04:40:01.648885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:01.649186 392 external_mini_cluster.cc:1428] Setting key cd62d51d9ffd85ce47883c14347c1dfc
I20260924 04:40:01.650866 21269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:01.651559 21269 log.cc:824] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:01.653725 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604: No bootstrap required, opened a new log
I20260924 04:40:01.654248 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:01.650459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57947 (local address 127.0.0.1:45807)
0924 04:40:01.650852 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:01.650866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:01.650899 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:40:01.651527 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:01.651542 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:01.651862 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:01.652226 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:01.653023 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:01.653027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:01.653037 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:01.653046 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:01.653094 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:01.653112 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:01.653258 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:01.653373 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 04:40:01.653525 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
I20260924 04:40:01.656808 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:01.657331 21269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206359a69cf6414a820b45da3301c604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45807 } }
I20260924 04:40:01.657526 21269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:01.657593 21269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206359a69cf6414a820b45da3301c604, State: Initialized, Role: FOLLOWER
I20260924 04:40:01.657984 21269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206359a69cf6414a820b45da3301c604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45807 } }
I20260924 04:40:01.658234 21269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:01.658318 21269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:01.658423 21269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:01.659447 21269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206359a69cf6414a820b45da3301c604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45807 } }
I20260924 04:40:01.659771 21269 leader_election.cc:302] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 206359a69cf6414a820b45da3301c604; no voters:
I20260924 04:40:01.660051 21269 leader_election.cc:288] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:01.660348 21279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:01.660737 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:01.658985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42063 (local address 127.0.0.1:45807)
0924 04:40:01.659092 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:01.659097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:01.659303 (+ 206us) server_negotiation.cc:407] Connection header received
0924 04:40:01.659444 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:01.659447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:01.659476 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:01.659573 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:01.660253 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:01.660259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:01.660262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:01.660275 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:01.660301 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:01.660314 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:01.660394 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:01.660563 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:40:01.660598 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:40:01.660609 21279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [term 1 LEADER]: Becoming Leader. State: Replica: 206359a69cf6414a820b45da3301c604, State: Running, Role: LEADER
I20260924 04:40:01.661195 21269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:01.661417 21279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206359a69cf6414a820b45da3301c604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45807 } }
I20260924 04:40:01.662454 21279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 206359a69cf6414a820b45da3301c604. Latest consensus state: current_term: 1 leader_uuid: "206359a69cf6414a820b45da3301c604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206359a69cf6414a820b45da3301c604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45807 } } }
I20260924 04:40:01.662638 21279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:01.662900 21280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "206359a69cf6414a820b45da3301c604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206359a69cf6414a820b45da3301c604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45807 } } }
I20260924 04:40:01.662993 21280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:01.663184 21291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:01.666296 21291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:01.671541 21291 catalog_manager.cc:1347] Generated new cluster ID: ffbc99d91b754e70bf91f56436e68c9b
I20260924 04:40:01.671627 21291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:01.681568 21291 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:01.681835 21291 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:01.696710 21291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 206359a69cf6414a820b45da3301c604: Generated new TSK 0
I20260924 04:40:01.697351 21291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:01.702822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 21201
2026-09-24T04:40:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:01 dist-test-slave-jlnt krb5kdc[21300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:01 dist-test-slave-jlnt krb5kdc[21300](info): set up 2 sockets
Sep 24 04:40:01 dist-test-slave-jlnt krb5kdc[21300](info): commencing operation
krb5kdc: starting...
W20260924 04:40:02.729269 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:02 dist-test-slave-jlnt krb5kdc[21300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:40:02.845050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:02.949819 21313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:02.950101 21313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:02.950179 21313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:02.954144 21313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:02.954219 21313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:02.954257 21313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:02.954313 21313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:02.954352 21313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:02.959378 21313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:02.960312 21313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:02.961387 21313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:02.969357 21319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:02.969360 21318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:02.969385 21321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:02.969645 21313 server_base.cc:1034] running on GCE node
I20260924 04:40:02.970248 21313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:02.971227 21313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:02.972414 21313 hybrid_clock.cc:648] HybridClock initialized: now 1790224802972396 us; error 30 us; skew 500 ppm
Sep 24 04:40:02 dist-test-slave-jlnt krb5kdc[21300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:02.974980 21313 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:40:02.975821 21313 webserver.cc:460] Webserver started at http://127.0.0.1:35855/ using document root <none> and password file <none>
I20260924 04:40:02.976351 21313 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:02.976437 21313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:02.976665 21313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:02.978605 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "1796803c4dc84411a0e34119a489a9a2"
format_stamp: "Formatted at 2026-09-24 04:40:02 on dist-test-slave-jlnt"
server_key: "755f6e58deb2ccbfc18ded6d8f98bd1e"
server_key_iv: "77c23f2ce21441103f5d5eaf1bcdb9ab"
server_key_version: "encryptionkey@0"
I20260924 04:40:02.979146 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "1796803c4dc84411a0e34119a489a9a2"
format_stamp: "Formatted at 2026-09-24 04:40:02 on dist-test-slave-jlnt"
server_key: "755f6e58deb2ccbfc18ded6d8f98bd1e"
server_key_iv: "77c23f2ce21441103f5d5eaf1bcdb9ab"
server_key_version: "encryptionkey@0"
I20260924 04:40:02.982652 21313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:40:02.984833 21328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:02.986003 21313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:40:02.986150 21313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "1796803c4dc84411a0e34119a489a9a2"
format_stamp: "Formatted at 2026-09-24 04:40:02 on dist-test-slave-jlnt"
server_key: "755f6e58deb2ccbfc18ded6d8f98bd1e"
server_key_iv: "77c23f2ce21441103f5d5eaf1bcdb9ab"
server_key_version: "encryptionkey@0"
I20260924 04:40:02.986259 21313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:02.996932 21313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:02.999831 21313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:03.000044 21313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:03.007288 21313 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40347
I20260924 04:40:03.007318 21380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40347 every 8 connection(s)
I20260924 04:40:03.008203 21313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:03.010123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 21313
I20260924 04:40:03.010269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:03.010567 392 external_mini_cluster.cc:1428] Setting key 5f754472f498e695eba7c747a5b29734
I20260924 04:40:03.011149 21381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:03.016667 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2: Bootstrap starting.
I20260924 04:40:03.019090 21381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2: Neither blocks nor log segments found. Creating new log.
Sep 24 04:40:03 dist-test-slave-jlnt krb5kdc[21300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:40:03.019942 21381 log.cc:824] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:03.022080 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2: No bootstrap required, opened a new log
I20260924 04:40:03.025025 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:03.011887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:43805 (local address 127.0.0.1:40347)
0924 04:40:03.012298 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:03.012311 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:03.012345 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:40:03.012960 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:03.012975 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:03.013382 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:03.013714 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:03.014532 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:03.015513 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:03.016367 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:03.016883 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:03.017048 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:03.019800 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:03.019805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:03.019817 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:03.019842 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:03.021940 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:03.022469 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:03.022475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:03.022480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:03.022623 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:03.022956 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:03.022959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:03.022960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:03.023420 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:03.023644 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:03.023868 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 04:40:03.024125 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":130,"threads_started":1}
I20260924 04:40:03.026393 21381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1796803c4dc84411a0e34119a489a9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } }
I20260924 04:40:03.026594 21381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:03.026662 21381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1796803c4dc84411a0e34119a489a9a2, State: Initialized, Role: FOLLOWER
I20260924 04:40:03.027104 21381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1796803c4dc84411a0e34119a489a9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } }
I20260924 04:40:03.027252 21381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:03.027340 21381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:03.027441 21381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:03.027985 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:03.028446 21381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1796803c4dc84411a0e34119a489a9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } }
I20260924 04:40:03.028765 21381 leader_election.cc:302] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1796803c4dc84411a0e34119a489a9a2; no voters:
I20260924 04:40:03.029045 21381 leader_election.cc:288] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:03.029390 21390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:03.029597 21390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1796803c4dc84411a0e34119a489a9a2, State: Running, Role: LEADER
I20260924 04:40:03.029942 21390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1796803c4dc84411a0e34119a489a9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } }
I20260924 04:40:03.030305 21381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:03.031740 21392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1796803c4dc84411a0e34119a489a9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1796803c4dc84411a0e34119a489a9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } } }
I20260924 04:40:03.031798 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1796803c4dc84411a0e34119a489a9a2. Latest consensus state: current_term: 1 leader_uuid: "1796803c4dc84411a0e34119a489a9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1796803c4dc84411a0e34119a489a9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } } }
I20260924 04:40:03.031895 21392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:03.031917 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:03.032284 21403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:03.035275 21403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:03.038166 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:03.029953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:40575 (local address 127.0.0.1:40347)
0924 04:40:03.030051 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:03.030055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:03.030321 (+ 266us) server_negotiation.cc:407] Connection header received
0924 04:40:03.030473 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:03.030476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:03.030514 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:03.030603 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:03.031304 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:03.032175 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:03.033218 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:03.033464 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:03.033571 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:03.035082 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:03.035086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:03.035088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:03.035111 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:03.036429 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:03.036880 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:03.036884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:03.036886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:03.036973 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:03.037321 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:03.037324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:03.037325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:03.037491 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:03.037638 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:03.037897 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 04:40:03.037995 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:40:03.040704 21403 catalog_manager.cc:1347] Generated new cluster ID: b6150062bf2a42cd8ff91b7ca65c7b4b
I20260924 04:40:03.040776 21403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:03.053639 21403 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:03.054586 21403 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:03.059767 21403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1796803c4dc84411a0e34119a489a9a2: Generated new TSK 0
I20260924 04:40:03.060334 21403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:03.065419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 21313
W20260924 04:40:03.070535 21298 connection.cc:535] client connection to 127.0.0.1:40347 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40347): Connection reset by peer (error 104)
2026-09-24T04:40:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:03 dist-test-slave-jlnt krb5kdc[21412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:03 dist-test-slave-jlnt krb5kdc[21412](info): set up 2 sockets
Sep 24 04:40:03 dist-test-slave-jlnt krb5kdc[21412](info): commencing operation
krb5kdc: starting...
W20260924 04:40:04.101454 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:04 dist-test-slave-jlnt krb5kdc[21412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:40:04.217538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:04.320520 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:04.320780 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:04.320851 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:04.324704 21425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:04.324782 21425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:04.324819 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:04.324877 21425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:04.324916 21425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:04.329890 21425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:04.330770 21425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:04.331670 21425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:04.339373 21433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:04.339419 21431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:04.339378 21430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:04.339558 21425 server_base.cc:1034] running on GCE node
I20260924 04:40:04.340166 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:04.341110 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:04.342326 21425 hybrid_clock.cc:648] HybridClock initialized: now 1790224804342300 us; error 37 us; skew 500 ppm
Sep 24 04:40:04 dist-test-slave-jlnt krb5kdc[21412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:04.353870 21425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:40:04.354672 21425 webserver.cc:460] Webserver started at http://127.0.0.1:43779/ using document root <none> and password file <none>
I20260924 04:40:04.355135 21425 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:04.355190 21425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:04.355338 21425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:04.357182 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "a47fc21c05f243b0bfc69042f041977c"
format_stamp: "Formatted at 2026-09-24 04:40:04 on dist-test-slave-jlnt"
server_key: "4a32b59ac8b65906d5a74a2e818afd34"
server_key_iv: "8b0937ad5669a39929f70236aed25050"
server_key_version: "encryptionkey@0"
I20260924 04:40:04.357721 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "a47fc21c05f243b0bfc69042f041977c"
format_stamp: "Formatted at 2026-09-24 04:40:04 on dist-test-slave-jlnt"
server_key: "4a32b59ac8b65906d5a74a2e818afd34"
server_key_iv: "8b0937ad5669a39929f70236aed25050"
server_key_version: "encryptionkey@0"
I20260924 04:40:04.360988 21425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 04:40:04.363215 21440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:04.364342 21425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:40:04.364434 21425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "a47fc21c05f243b0bfc69042f041977c"
format_stamp: "Formatted at 2026-09-24 04:40:04 on dist-test-slave-jlnt"
server_key: "4a32b59ac8b65906d5a74a2e818afd34"
server_key_iv: "8b0937ad5669a39929f70236aed25050"
server_key_version: "encryptionkey@0"
I20260924 04:40:04.364517 21425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:04.374902 21425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:04.377504 21425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:04.377635 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:04.384414 21425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37427
I20260924 04:40:04.384444 21492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37427 every 8 connection(s)
I20260924 04:40:04.385281 21425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:04.387887 21493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:04.392616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 21425
I20260924 04:40:04.392781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:04.393101 392 external_mini_cluster.cc:1428] Setting key 60189fb0e29c732cff8d6004aba0d71e
I20260924 04:40:04.393747 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c: Bootstrap starting.
I20260924 04:40:04.396205 21493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:04.396966 21493 log.cc:824] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:04.398913 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c: No bootstrap required, opened a new log
I20260924 04:40:04.402040 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:04.394370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:58571 (local address 127.0.0.1:37427)
0924 04:40:04.394740 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:04.394755 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:04.394788 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:40:04.395367 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:04.395383 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:04.395706 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:04.395964 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:04.396397 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:04.397443 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:04.398449 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:04.399103 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:04.399259 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:04.400072 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:04.400078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:04.400092 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:04.400548 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:04.400614 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:04.400641 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:04.400777 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:04.400940 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 04:40:04.401070 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":119,"threads_started":1}
I20260924 04:40:04.403352 21493 raft_consensus.cc:348] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47fc21c05f243b0bfc69042f041977c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260924 04:40:04.403585 21493 raft_consensus.cc:374] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:04.403653 21493 raft_consensus.cc:729] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47fc21c05f243b0bfc69042f041977c, State: Initialized, Role: FOLLOWER
I20260924 04:40:04.404090 21493 consensus_queue.cc:260] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47fc21c05f243b0bfc69042f041977c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260924 04:40:04.404258 21493 raft_consensus.cc:388] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:04.404328 21493 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:04.404420 21493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:04.405090 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:04.405308 21493 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47fc21c05f243b0bfc69042f041977c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260924 04:40:04.405644 21493 leader_election.cc:302] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a47fc21c05f243b0bfc69042f041977c; no voters:
I20260924 04:40:04.405957 21493 leader_election.cc:288] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:04.406064 21498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:04.406309 21498 raft_consensus.cc:686] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [term 1 LEADER]: Becoming Leader. State: Replica: a47fc21c05f243b0bfc69042f041977c, State: Running, Role: LEADER
I20260924 04:40:04.406684 21498 consensus_queue.cc:237] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47fc21c05f243b0bfc69042f041977c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260924 04:40:04.407146 21493 sys_catalog.cc:564] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:04.408411 21500 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47fc21c05f243b0bfc69042f041977c. Latest consensus state: current_term: 1 leader_uuid: "a47fc21c05f243b0bfc69042f041977c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47fc21c05f243b0bfc69042f041977c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } } }
I20260924 04:40:04.408543 21500 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:04.408758 21499 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a47fc21c05f243b0bfc69042f041977c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47fc21c05f243b0bfc69042f041977c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } } }
I20260924 04:40:04.408900 21499 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c [sys.catalog]: This master's current role is: LEADER
W20260924 04:40:04.412034 21517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:04.412122 21517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:40:04.412196 21518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:04.413204 21518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:04.414480 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:04.407263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:53749 (local address 127.0.0.1:37427)
0924 04:40:04.407354 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:04.407358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:04.409715 (+ 2357us) server_negotiation.cc:407] Connection header received
0924 04:40:04.409837 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:04.409841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:04.409888 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:04.409973 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:04.410207 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:04.411043 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:04.412265 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:04.412598 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:04.413121 (+ 523us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:04.413701 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:04.413705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:04.413708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:04.413721 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:04.413748 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:04.413762 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:04.414260 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:04.414318 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 04:40:04.414350 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:40:04.418913 21518 catalog_manager.cc:1347] Generated new cluster ID: 7e7e459c0d594948af82c30cf6e15549
I20260924 04:40:04.418984 21518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:04.439045 21518 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:04.440016 21518 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:04.446327 21518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a47fc21c05f243b0bfc69042f041977c: Generated new TSK 0
I20260924 04:40:04.446952 21518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:04.452474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 21425
W20260924 04:40:04.457964 21410 connection.cc:535] client connection to 127.0.0.1:37427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37427): Connection reset by peer (error 104)
2026-09-24T04:40:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:04 dist-test-slave-jlnt krb5kdc[21524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:04 dist-test-slave-jlnt krb5kdc[21524](info): set up 2 sockets
Sep 24 04:40:04 dist-test-slave-jlnt krb5kdc[21524](info): commencing operation
krb5kdc: starting...
W20260924 04:40:05.485968 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:05 dist-test-slave-jlnt krb5kdc[21524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:40:05.602054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:05.706249 21537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:05.706471 21537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:05.706512 21537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:05.710417 21537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:05.710469 21537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:05.710489 21537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:05.710505 21537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:05.710559 21537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:05.715540 21537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:05.716532 21537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:05.717480 21537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:05.725401 21542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:05.725461 21545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:05.725476 21543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:05.725694 21537 server_base.cc:1034] running on GCE node
I20260924 04:40:05.726187 21537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:05.727180 21537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:05.728333 21537 hybrid_clock.cc:648] HybridClock initialized: now 1790224805728304 us; error 41 us; skew 500 ppm
Sep 24 04:40:05 dist-test-slave-jlnt krb5kdc[21524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:05.731014 21537 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:40:05.731832 21537 webserver.cc:460] Webserver started at http://127.0.0.1:40163/ using document root <none> and password file <none>
I20260924 04:40:05.732343 21537 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:05.732400 21537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:05.732632 21537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:05.734563 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "0532806149e84b5a884c40c32acab8dd"
format_stamp: "Formatted at 2026-09-24 04:40:05 on dist-test-slave-jlnt"
server_key: "c11cba82834280d732073df6fdb1a700"
server_key_iv: "7c0d75e56f4ee4213a51c403c7aa55ac"
server_key_version: "encryptionkey@0"
I20260924 04:40:05.735100 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "0532806149e84b5a884c40c32acab8dd"
format_stamp: "Formatted at 2026-09-24 04:40:05 on dist-test-slave-jlnt"
server_key: "c11cba82834280d732073df6fdb1a700"
server_key_iv: "7c0d75e56f4ee4213a51c403c7aa55ac"
server_key_version: "encryptionkey@0"
I20260924 04:40:05.738693 21537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 04:40:05.740912 21552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:05.742092 21537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 04:40:05.742226 21537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "0532806149e84b5a884c40c32acab8dd"
format_stamp: "Formatted at 2026-09-24 04:40:05 on dist-test-slave-jlnt"
server_key: "c11cba82834280d732073df6fdb1a700"
server_key_iv: "7c0d75e56f4ee4213a51c403c7aa55ac"
server_key_version: "encryptionkey@0"
I20260924 04:40:05.742331 21537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:05.746589 21537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:05.749436 21537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:05.749572 21537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:05.756147 21537 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42043
I20260924 04:40:05.756160 21604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42043 every 8 connection(s)
I20260924 04:40:05.757036 21537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:05.759680 21605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:05.765007 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd: Bootstrap starting.
I20260924 04:40:05.767156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 21537
I20260924 04:40:05.767328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:05.767405 21605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:05.767623 392 external_mini_cluster.cc:1428] Setting key eb3690a8a968aafd182d17dcd79b8d2a
I20260924 04:40:05.768234 21605 log.cc:824] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:05.770305 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd: No bootstrap required, opened a new log
I20260924 04:40:05.772727 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:05.768731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:49249 (local address 127.0.0.1:42043)
0924 04:40:05.769114 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:05.769132 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:05.769192 (+ 60us) server_negotiation.cc:407] Connection header received
0924 04:40:05.770006 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:05.770029 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:05.770360 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:05.770709 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:05.771402 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:05.771408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:05.771422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:05.771437 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:05.771495 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:05.771519 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:05.771648 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:05.771746 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 04:40:05.771893 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260924 04:40:05.774641 21605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0532806149e84b5a884c40c32acab8dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } }
I20260924 04:40:05.774864 21605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:05.774961 21605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0532806149e84b5a884c40c32acab8dd, State: Initialized, Role: FOLLOWER
I20260924 04:40:05.775358 21605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0532806149e84b5a884c40c32acab8dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } }
I20260924 04:40:05.775506 21605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:05.775576 21605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:05.775591 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:05.775676 21605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:05.776516 21605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0532806149e84b5a884c40c32acab8dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } }
I20260924 04:40:05.776872 21605 leader_election.cc:302] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0532806149e84b5a884c40c32acab8dd; no voters:
I20260924 04:40:05.777232 21605 leader_election.cc:288] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:05.777288 21614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:05.777522 21614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [term 1 LEADER]: Becoming Leader. State: Replica: 0532806149e84b5a884c40c32acab8dd, State: Running, Role: LEADER
I20260924 04:40:05.777839 21614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0532806149e84b5a884c40c32acab8dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } }
I20260924 04:40:05.778445 21605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:05.779055 21614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0532806149e84b5a884c40c32acab8dd. Latest consensus state: current_term: 1 leader_uuid: "0532806149e84b5a884c40c32acab8dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0532806149e84b5a884c40c32acab8dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } } }
I20260924 04:40:05.779179 21614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:05.779618 21621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:05.779608 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0532806149e84b5a884c40c32acab8dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0532806149e84b5a884c40c32acab8dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } } }
I20260924 04:40:05.779803 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:05.781574 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:05.777948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:51845 (local address 127.0.0.1:42043)
0924 04:40:05.778040 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:05.778044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:05.779622 (+ 1578us) server_negotiation.cc:407] Connection header received
0924 04:40:05.779810 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:05.779816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:05.779853 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:05.779905 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:05.781096 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:05.781099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:05.781102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:05.781113 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:05.781157 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:05.781170 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:05.781223 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:05.781419 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 04:40:05.781460 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:40:05.782773 21621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:05.787552 21621 catalog_manager.cc:1347] Generated new cluster ID: 2865fb02936f4bddbfd5e10f981db5d8
I20260924 04:40:05.787607 21621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:05.802274 21621 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:05.802536 21621 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:05.813552 21621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0532806149e84b5a884c40c32acab8dd: Generated new TSK 0
I20260924 04:40:05.814344 21621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:05.817497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 21537
2026-09-24T04:40:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:05 dist-test-slave-jlnt krb5kdc[21636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:05 dist-test-slave-jlnt krb5kdc[21636](info): set up 2 sockets
Sep 24 04:40:05 dist-test-slave-jlnt krb5kdc[21636](info): commencing operation
krb5kdc: starting...
W20260924 04:40:06.849362 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:06 dist-test-slave-jlnt krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:40:06.967941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:07.072772 21649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:07.073000 21649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:07.073038 21649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:07.076910 21649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:07.076961 21649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:07.076980 21649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:07.076996 21649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:07.077011 21649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:07.081979 21649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:07.082902 21649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:07.083832 21649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:07.091894 21655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:07.091908 21654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:07.091940 21649 server_base.cc:1034] running on GCE node
W20260924 04:40:07.091897 21657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:07.092630 21649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:07.093655 21649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:07.094815 21649 hybrid_clock.cc:648] HybridClock initialized: now 1790224807094788 us; error 40 us; skew 500 ppm
Sep 24 04:40:07 dist-test-slave-jlnt krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:07.097410 21649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:40:07.098265 21649 webserver.cc:460] Webserver started at http://127.0.0.1:34895/ using document root <none> and password file <none>
I20260924 04:40:07.098793 21649 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:07.098850 21649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:07.099108 21649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:07.101045 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "972a868ac05b4bdd9d794574a98f7a6b"
format_stamp: "Formatted at 2026-09-24 04:40:07 on dist-test-slave-jlnt"
server_key: "f74b0f25ea2e580960c8f348a54d0a55"
server_key_iv: "d47f081c2384ad49810efb9e19b50ce2"
server_key_version: "encryptionkey@0"
I20260924 04:40:07.101639 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "972a868ac05b4bdd9d794574a98f7a6b"
format_stamp: "Formatted at 2026-09-24 04:40:07 on dist-test-slave-jlnt"
server_key: "f74b0f25ea2e580960c8f348a54d0a55"
server_key_iv: "d47f081c2384ad49810efb9e19b50ce2"
server_key_version: "encryptionkey@0"
I20260924 04:40:07.105084 21649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:40:07.107380 21664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:07.108682 21649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:40:07.108774 21649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "972a868ac05b4bdd9d794574a98f7a6b"
format_stamp: "Formatted at 2026-09-24 04:40:07 on dist-test-slave-jlnt"
server_key: "f74b0f25ea2e580960c8f348a54d0a55"
server_key_iv: "d47f081c2384ad49810efb9e19b50ce2"
server_key_version: "encryptionkey@0"
I20260924 04:40:07.108851 21649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:07.127317 21649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:07.130419 21649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:07.130558 21649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:07.136686 21649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35717
I20260924 04:40:07.136776 21716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35717 every 8 connection(s)
I20260924 04:40:07.137630 21649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:07.140183 21717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:07.143273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 21649
I20260924 04:40:07.143443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:07.143716 392 external_mini_cluster.cc:1428] Setting key dd61250fc00472234ae2d9628f67207f
I20260924 04:40:07.145750 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b: Bootstrap starting.
I20260924 04:40:07.148226 21717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:07.149015 21717 log.cc:824] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:07.150946 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b: No bootstrap required, opened a new log
Sep 24 04:40:07 dist-test-slave-jlnt krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:40:07.154817 21717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972a868ac05b4bdd9d794574a98f7a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260924 04:40:07.155076 21717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:07.155128 21717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972a868ac05b4bdd9d794574a98f7a6b, State: Initialized, Role: FOLLOWER
I20260924 04:40:07.155582 21717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972a868ac05b4bdd9d794574a98f7a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260924 04:40:07.155737 21717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:07.155804 21717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:07.155915 21717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:07.156984 21717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972a868ac05b4bdd9d794574a98f7a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260924 04:40:07.157351 21717 leader_election.cc:302] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 972a868ac05b4bdd9d794574a98f7a6b; no voters:
I20260924 04:40:07.157639 21717 leader_election.cc:288] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:07.157727 21722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:07.157944 21722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [term 1 LEADER]: Becoming Leader. State: Replica: 972a868ac05b4bdd9d794574a98f7a6b, State: Running, Role: LEADER
I20260924 04:40:07.158264 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:07.144903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:33741 (local address 127.0.0.1:35717)
0924 04:40:07.145308 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:07.145325 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:07.145374 (+ 49us) server_negotiation.cc:407] Connection header received
0924 04:40:07.146162 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:07.146183 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:07.146573 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:07.146935 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:07.147564 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:07.149020 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:07.149905 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:07.150541 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:07.150688 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:07.153071 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:07.153080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:07.153098 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:07.153157 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:07.155062 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:07.155641 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:07.155646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:07.155652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:07.155786 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:07.156186 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:07.156192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:07.156196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:07.156649 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:07.156878 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:07.157070 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 04:40:07.157302 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":110,"threads_started":1}
I20260924 04:40:07.158317 21722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972a868ac05b4bdd9d794574a98f7a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260924 04:40:07.158790 21717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:07.159446 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 972a868ac05b4bdd9d794574a98f7a6b. Latest consensus state: current_term: 1 leader_uuid: "972a868ac05b4bdd9d794574a98f7a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972a868ac05b4bdd9d794574a98f7a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } } }
I20260924 04:40:07.159618 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:07.161366 21723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "972a868ac05b4bdd9d794574a98f7a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972a868ac05b4bdd9d794574a98f7a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } } }
I20260924 04:40:07.161469 21723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:07.161476 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 04:40:07.165040 21737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:07.165124 21737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:40:07.165251 21731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:07.166105 21731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:07.172127 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:07.163375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:37213 (local address 127.0.0.1:35717)
0924 04:40:07.163482 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:07.163486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:07.163635 (+ 149us) server_negotiation.cc:407] Connection header received
0924 04:40:07.163743 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:07.163747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:07.163793 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:07.163866 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:07.164606 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:07.165352 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:07.166415 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:07.166764 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:07.166871 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:07.168313 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:07.168316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:07.168319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:07.168342 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:07.170082 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:07.170712 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:07.170715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:07.170717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:07.170802 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:07.171208 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:07.171212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:07.171214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:07.171419 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:07.171579 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:07.171876 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 04:40:07.171973 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:40:07.178623 21731 catalog_manager.cc:1347] Generated new cluster ID: efbf508be9b84554b84c9e45ee488b2b
I20260924 04:40:07.178699 21731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:07.185894 21731 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:07.186969 21731 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:07.194562 21731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 972a868ac05b4bdd9d794574a98f7a6b: Generated new TSK 0
I20260924 04:40:07.195240 21731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:07.198649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 21649
W20260924 04:40:07.204283 21634 connection.cc:535] client connection to 127.0.0.1:35717 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35717): Connection reset by peer (error 104)
2026-09-24T04:40:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:07 dist-test-slave-jlnt krb5kdc[21748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:07 dist-test-slave-jlnt krb5kdc[21748](info): set up 2 sockets
Sep 24 04:40:07 dist-test-slave-jlnt krb5kdc[21748](info): commencing operation
krb5kdc: starting...
W20260924 04:40:08.233319 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:08 dist-test-slave-jlnt krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:40:08.348330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:08.454854 21761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:08.455120 21761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:08.455199 21761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:08.459169 21761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:08.459254 21761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:08.459292 21761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:08.459354 21761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:08.459394 21761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:08.464498 21761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:08.465547 21761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:08.466549 21761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:08.474707 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:08.474809 21761 server_base.cc:1034] running on GCE node
W20260924 04:40:08.474711 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:08.474722 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:08.475471 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:08.476495 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:08.477684 21761 hybrid_clock.cc:648] HybridClock initialized: now 1790224808477677 us; error 28 us; skew 500 ppm
Sep 24 04:40:08 dist-test-slave-jlnt krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:08.480206 21761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:40:08.481009 21761 webserver.cc:460] Webserver started at http://127.0.0.1:34995/ using document root <none> and password file <none>
I20260924 04:40:08.481595 21761 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:08.481685 21761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:08.481902 21761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:08.483858 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "cba6746fe6f64aa391a04f538b60554b"
format_stamp: "Formatted at 2026-09-24 04:40:08 on dist-test-slave-jlnt"
server_key: "c29500425b7587b51f024298817209ed"
server_key_iv: "1bf81034c215e08df198b8759c442f21"
server_key_version: "encryptionkey@0"
I20260924 04:40:08.484393 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "cba6746fe6f64aa391a04f538b60554b"
format_stamp: "Formatted at 2026-09-24 04:40:08 on dist-test-slave-jlnt"
server_key: "c29500425b7587b51f024298817209ed"
server_key_iv: "1bf81034c215e08df198b8759c442f21"
server_key_version: "encryptionkey@0"
I20260924 04:40:08.487831 21761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:40:08.490168 21776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:08.491305 21761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:40:08.491436 21761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "cba6746fe6f64aa391a04f538b60554b"
format_stamp: "Formatted at 2026-09-24 04:40:08 on dist-test-slave-jlnt"
server_key: "c29500425b7587b51f024298817209ed"
server_key_iv: "1bf81034c215e08df198b8759c442f21"
server_key_version: "encryptionkey@0"
I20260924 04:40:08.491552 21761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:08.512667 21761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:08.515396 21761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:08.515594 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:08.522588 21761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34549
I20260924 04:40:08.522601 21828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34549 every 8 connection(s)
I20260924 04:40:08.523548 21761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:08.526180 21829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:08.531340 21829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b: Bootstrap starting.
I20260924 04:40:08.533586 21829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:08.533762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 21761
I20260924 04:40:08.533905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:08.534181 392 external_mini_cluster.cc:1428] Setting key e8bf2a68715fad9f352868b2ab5823c7
I20260924 04:40:08.534421 21829 log.cc:824] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:08.536289 21829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b: No bootstrap required, opened a new log
I20260924 04:40:08.540131 21829 raft_consensus.cc:348] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba6746fe6f64aa391a04f538b60554b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } }
I20260924 04:40:08.540395 21829 raft_consensus.cc:374] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:08.540458 21829 raft_consensus.cc:729] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba6746fe6f64aa391a04f538b60554b, State: Initialized, Role: FOLLOWER
I20260924 04:40:08.540987 21829 consensus_queue.cc:260] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba6746fe6f64aa391a04f538b60554b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } }
I20260924 04:40:08.541186 21829 raft_consensus.cc:388] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:08.541256 21829 raft_consensus.cc:482] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:08.541355 21829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:08.542327 21829 raft_consensus.cc:504] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba6746fe6f64aa391a04f538b60554b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } }
I20260924 04:40:08.542644 21829 leader_election.cc:302] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cba6746fe6f64aa391a04f538b60554b; no voters:
I20260924 04:40:08.542913 21829 leader_election.cc:288] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:08.542954 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:08.535251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:44559 (local address 127.0.0.1:34549)
0924 04:40:08.535669 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:08.535692 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:08.535727 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:40:08.536352 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:08.536368 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:08.536733 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:08.537004 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:08.537362 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:08.538268 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:08.539571 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:08.540056 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:08.540172 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:08.541080 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:08.541085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:08.541096 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:08.541630 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:08.541684 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:08.541706 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:08.541805 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:08.541988 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:40:08.542147 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260924 04:40:08.543022 21834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:08.543290 21834 raft_consensus.cc:686] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [term 1 LEADER]: Becoming Leader. State: Replica: cba6746fe6f64aa391a04f538b60554b, State: Running, Role: LEADER
I20260924 04:40:08.543632 21834 consensus_queue.cc:237] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba6746fe6f64aa391a04f538b60554b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } }
I20260924 04:40:08.544096 21829 sys_catalog.cc:564] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:08.544770 21834 sys_catalog.cc:455] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cba6746fe6f64aa391a04f538b60554b. Latest consensus state: current_term: 1 leader_uuid: "cba6746fe6f64aa391a04f538b60554b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba6746fe6f64aa391a04f538b60554b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } } }
I20260924 04:40:08.544893 21834 sys_catalog.cc:458] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:08.545351 21836 sys_catalog.cc:455] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cba6746fe6f64aa391a04f538b60554b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba6746fe6f64aa391a04f538b60554b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34549 } } }
I20260924 04:40:08.545492 21836 sys_catalog.cc:458] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:08.546118 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:08.546106 21843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:08.548758 21843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:08.553679 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:08.549326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:50441 (local address 127.0.0.1:34549)
0924 04:40:08.549423 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:08.549427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:08.549676 (+ 249us) server_negotiation.cc:407] Connection header received
0924 04:40:08.549776 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:08.549779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:08.549834 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:08.549915 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:08.550201 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:08.550884 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:08.551679 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:08.551919 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:08.552029 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:08.553180 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:08.553185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:08.553188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:08.553200 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:08.553225 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:08.553239 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:08.553337 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:08.553513 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 04:40:08.553547 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:40:08.554510 21843 catalog_manager.cc:1347] Generated new cluster ID: ef4347a8862743ca902863a80e9ca578
I20260924 04:40:08.554564 21843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:08.568418 21843 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:08.569563 21843 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:08.581276 21843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cba6746fe6f64aa391a04f538b60554b: Generated new TSK 0
I20260924 04:40:08.581907 21843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:08.593219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 21761
W20260924 04:40:08.597985 21746 connection.cc:535] client connection to 127.0.0.1:34549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34549): Connection reset by peer (error 104)
2026-09-24T04:40:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:08 dist-test-slave-jlnt krb5kdc[21860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:08 dist-test-slave-jlnt krb5kdc[21860](info): set up 2 sockets
Sep 24 04:40:08 dist-test-slave-jlnt krb5kdc[21860](info): commencing operation
krb5kdc: starting...
W20260924 04:40:09.649307 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:09 dist-test-slave-jlnt krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:40:09.766510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:09.872171 21873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:09.872435 21873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:09.872519 21873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:09.876405 21873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:09.876484 21873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:09.876520 21873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:09.876574 21873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:09.876612 21873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:09.881594 21873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:09.882520 21873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:09.883477 21873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:09.891355 21879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:09.891367 21881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:09.891367 21878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:09.891687 21873 server_base.cc:1034] running on GCE node
I20260924 04:40:09.892225 21873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:09.893267 21873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:09.894449 21873 hybrid_clock.cc:648] HybridClock initialized: now 1790224809894435 us; error 34 us; skew 500 ppm
Sep 24 04:40:09 dist-test-slave-jlnt krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:09.904093 21873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:40:09.904953 21873 webserver.cc:460] Webserver started at http://127.0.0.1:42039/ using document root <none> and password file <none>
I20260924 04:40:09.905562 21873 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:09.905652 21873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:09.905879 21873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:09.907851 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5"
format_stamp: "Formatted at 2026-09-24 04:40:09 on dist-test-slave-jlnt"
server_key: "ade13533ddfb877d4db07bca8fd8056f"
server_key_iv: "b9f65dcf8574648b609288d08adc6746"
server_key_version: "encryptionkey@0"
I20260924 04:40:09.908407 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5"
format_stamp: "Formatted at 2026-09-24 04:40:09 on dist-test-slave-jlnt"
server_key: "ade13533ddfb877d4db07bca8fd8056f"
server_key_iv: "b9f65dcf8574648b609288d08adc6746"
server_key_version: "encryptionkey@0"
I20260924 04:40:09.912084 21873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:40:09.914546 21888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:09.915668 21873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:40:09.915808 21873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5"
format_stamp: "Formatted at 2026-09-24 04:40:09 on dist-test-slave-jlnt"
server_key: "ade13533ddfb877d4db07bca8fd8056f"
server_key_iv: "b9f65dcf8574648b609288d08adc6746"
server_key_version: "encryptionkey@0"
I20260924 04:40:09.915915 21873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:09.920351 21873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:09.923146 21873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:09.923329 21873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:09.930611 21873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45775
I20260924 04:40:09.930636 21940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45775 every 8 connection(s)
I20260924 04:40:09.931532 21873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:09.934185 21941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:09.939404 21941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5: Bootstrap starting.
I20260924 04:40:09.941622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 21873
I20260924 04:40:09.941751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:09.941927 21941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:09.942062 392 external_mini_cluster.cc:1428] Setting key 87cb1f19f7d1ad57679a51e0a5f22f45
I20260924 04:40:09.942811 21941 log.cc:824] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:09.944694 21941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5: No bootstrap required, opened a new log
I20260924 04:40:09.946790 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:09.943143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:56671 (local address 127.0.0.1:45775)
0924 04:40:09.943609 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:09.943627 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:09.943669 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:40:09.944268 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:09.944283 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:09.944549 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:09.944880 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:09.945661 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:09.945666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:09.945680 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:09.945698 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:09.945753 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:09.945771 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:09.945880 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:09.945997 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 04:40:09.946145 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":126,"threads_started":1}
I20260924 04:40:09.948716 21941 raft_consensus.cc:348] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } }
I20260924 04:40:09.949025 21941 raft_consensus.cc:374] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:09.949106 21941 raft_consensus.cc:729] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01aadad9ec34c858e8c3f69cbcf7cf5, State: Initialized, Role: FOLLOWER
I20260924 04:40:09.949535 21941 consensus_queue.cc:260] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } }
I20260924 04:40:09.949661 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:09.949714 21941 raft_consensus.cc:388] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:09.949788 21941 raft_consensus.cc:482] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:09.949898 21941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:09.950730 21941 raft_consensus.cc:504] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } }
I20260924 04:40:09.951077 21941 leader_election.cc:302] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b01aadad9ec34c858e8c3f69cbcf7cf5; no voters:
I20260924 04:40:09.951363 21941 leader_election.cc:288] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:09.951464 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:09.951710 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [term 1 LEADER]: Becoming Leader. State: Replica: b01aadad9ec34c858e8c3f69cbcf7cf5, State: Running, Role: LEADER
I20260924 04:40:09.952159 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } }
I20260924 04:40:09.952656 21941 sys_catalog.cc:564] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:09.953464 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:09.951736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:51421 (local address 127.0.0.1:45775)
0924 04:40:09.951819 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:09.951824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:09.952051 (+ 227us) server_negotiation.cc:407] Connection header received
0924 04:40:09.952152 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:09.952155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:09.952188 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:09.952270 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:09.953011 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:09.953014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:09.953019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:09.953032 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:09.953065 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:09.953078 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:09.953189 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:09.953295 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 04:40:09.953330 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 04:40:09.953450 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } } }
I20260924 04:40:09.953823 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:09.953924 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b01aadad9ec34c858e8c3f69cbcf7cf5. Latest consensus state: current_term: 1 leader_uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01aadad9ec34c858e8c3f69cbcf7cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45775 } } }
I20260924 04:40:09.954146 21960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:09.954929 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:09.957509 21960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:09.962129 21960 catalog_manager.cc:1347] Generated new cluster ID: adf11f8c25fc428a9c34a35afa59487a
I20260924 04:40:09.962186 21960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:09.993544 21960 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:09.993878 21960 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:10.004959 21960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b01aadad9ec34c858e8c3f69cbcf7cf5: Generated new TSK 0
I20260924 04:40:10.005671 21960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:10.014091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 21873
2026-09-24T04:40:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:10 dist-test-slave-jlnt krb5kdc[21972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:10 dist-test-slave-jlnt krb5kdc[21972](info): set up 2 sockets
Sep 24 04:40:10 dist-test-slave-jlnt krb5kdc[21972](info): commencing operation
krb5kdc: starting...
W20260924 04:40:11.037228 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:11 dist-test-slave-jlnt krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:40:11.157300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:11.265940 21985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:11.266168 21985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:11.266208 21985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:11.270123 21985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:11.270169 21985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:11.270188 21985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:11.270242 21985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:11.270270 21985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:11.275305 21985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:11.276244 21985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:11.277200 21985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:40:11.285279 21985 server_base.cc:1034] running on GCE node
W20260924 04:40:11.285220 21993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:11.285230 21990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:11.285204 21991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:11.285997 21985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:11.287012 21985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:11.288200 21985 hybrid_clock.cc:648] HybridClock initialized: now 1790224811288188 us; error 24 us; skew 500 ppm
Sep 24 04:40:11 dist-test-slave-jlnt krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:11.290912 21985 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:40:11.291687 21985 webserver.cc:460] Webserver started at http://127.0.98.62:34281/ using document root <none> and password file <none>
I20260924 04:40:11.292171 21985 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:11.292227 21985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:11.292377 21985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:11.294312 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "6e2568d05cbb40f481b0e95ce226e11a"
format_stamp: "Formatted at 2026-09-24 04:40:11 on dist-test-slave-jlnt"
server_key: "8c3556d25fba821c92f356dc361d600c"
server_key_iv: "e9445f5e345e4af0d7e9387c8d0c0c31"
server_key_version: "encryptionkey@0"
I20260924 04:40:11.294806 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e2568d05cbb40f481b0e95ce226e11a"
format_stamp: "Formatted at 2026-09-24 04:40:11 on dist-test-slave-jlnt"
server_key: "8c3556d25fba821c92f356dc361d600c"
server_key_iv: "e9445f5e345e4af0d7e9387c8d0c0c31"
server_key_version: "encryptionkey@0"
I20260924 04:40:11.298237 21985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:40:11.300426 22000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:11.301628 21985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:40:11.301723 21985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "6e2568d05cbb40f481b0e95ce226e11a"
format_stamp: "Formatted at 2026-09-24 04:40:11 on dist-test-slave-jlnt"
server_key: "8c3556d25fba821c92f356dc361d600c"
server_key_iv: "e9445f5e345e4af0d7e9387c8d0c0c31"
server_key_version: "encryptionkey@0"
I20260924 04:40:11.301801 21985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:11.328857 21985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:11.331831 21985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:11.331992 21985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:11.339030 21985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36287
I20260924 04:40:11.339046 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36287 every 8 connection(s)
I20260924 04:40:11.339954 21985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:11.342656 22053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:11.342684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 21985
I20260924 04:40:11.342868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:11.343207 392 external_mini_cluster.cc:1428] Setting key a61f7cf87590a836b8d97cf61c374a26
I20260924 04:40:11.349753 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a: Bootstrap starting.
I20260924 04:40:11.352396 22053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a: Neither blocks nor log segments found. Creating new log.
Sep 24 04:40:11 dist-test-slave-jlnt krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:40:11.353190 22053 log.cc:824] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:11.355436 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a: No bootstrap required, opened a new log
I20260924 04:40:11.358775 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:11.344615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35963 (local address 127.0.98.62:36287)
0924 04:40:11.345071 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:11.345086 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:11.345123 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:40:11.345927 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:11.345944 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:11.346288 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:11.346585 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:11.347516 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:11.348615 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:11.349838 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:11.350400 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:11.350528 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:11.353475 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:11.353481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:11.353500 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:11.353535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:11.355555 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:11.356067 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:11.356073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:11.356079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:11.356213 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:11.356533 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:11.356536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:11.356538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:11.356999 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:11.357220 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:11.357605 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 04:40:11.357868 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":136,"threads_started":1}
I20260924 04:40:11.359836 22053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2568d05cbb40f481b0e95ce226e11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } }
I20260924 04:40:11.360074 22053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:11.360146 22053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2568d05cbb40f481b0e95ce226e11a, State: Initialized, Role: FOLLOWER
I20260924 04:40:11.360638 22053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2568d05cbb40f481b0e95ce226e11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } }
I20260924 04:40:11.360826 22053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:11.360896 22053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:11.360988 22053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:11.361728 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:11.361882 22053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2568d05cbb40f481b0e95ce226e11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } }
I20260924 04:40:11.362208 22053 leader_election.cc:302] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e2568d05cbb40f481b0e95ce226e11a; no voters:
I20260924 04:40:11.362526 22053 leader_election.cc:288] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:11.362586 22058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:11.362787 22058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [term 1 LEADER]: Becoming Leader. State: Replica: 6e2568d05cbb40f481b0e95ce226e11a, State: Running, Role: LEADER
I20260924 04:40:11.363125 22058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2568d05cbb40f481b0e95ce226e11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } }
I20260924 04:40:11.363740 22053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:11.364714 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e2568d05cbb40f481b0e95ce226e11a. Latest consensus state: current_term: 1 leader_uuid: "6e2568d05cbb40f481b0e95ce226e11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2568d05cbb40f481b0e95ce226e11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } } }
I20260924 04:40:11.364843 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:11.365190 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e2568d05cbb40f481b0e95ce226e11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2568d05cbb40f481b0e95ce226e11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } } }
I20260924 04:40:11.365271 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:11.365262 22071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:11.368392 22071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:11.374511 22071 catalog_manager.cc:1347] Generated new cluster ID: 282bfad61c294fe49965c9432aede8de
I20260924 04:40:11.374569 22071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:11.375245 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:11.366173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32941 (local address 127.0.98.62:36287)
0924 04:40:11.366293 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:11.366298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:11.366715 (+ 417us) server_negotiation.cc:407] Connection header received
0924 04:40:11.366943 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:11.366946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:11.366988 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:11.367081 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:11.367806 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:11.368591 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:11.369678 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:11.369939 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:11.370075 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:11.371616 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:11.371620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:11.371622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:11.371646 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:11.373401 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:11.373878 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:11.373882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:11.373883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:11.373962 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:11.374333 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:11.374336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:11.374338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:11.374577 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:11.374722 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:11.375011 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 04:40:11.375109 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:40:11.394388 22071 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:11.395278 22071 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:11.401816 22071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e2568d05cbb40f481b0e95ce226e11a: Generated new TSK 0
I20260924 04:40:11.402477 22071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:11.408912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 21985
W20260924 04:40:11.414264 21970 connection.cc:535] client connection to 127.0.98.62:36287 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36287): Connection reset by peer (error 104)
2026-09-24T04:40:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:11 dist-test-slave-jlnt krb5kdc[22084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:11 dist-test-slave-jlnt krb5kdc[22084](info): set up 2 sockets
Sep 24 04:40:11 dist-test-slave-jlnt krb5kdc[22084](info): commencing operation
krb5kdc: starting...
W20260924 04:40:12.441377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:12 dist-test-slave-jlnt krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:40:12.563557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39035
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:12.668532 22097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:12.668761 22097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:12.668799 22097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:12.672721 22097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:12.672772 22097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:12.672791 22097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:12.672833 22097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:12.672868 22097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:12.677855 22097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:12.678794 22097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:12.679770 22097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:12.687908 22102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:12.687947 22103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:12.687995 22105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:12.688033 22097 server_base.cc:1034] running on GCE node
I20260924 04:40:12.688719 22097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:12.689857 22097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:12.691051 22097 hybrid_clock.cc:648] HybridClock initialized: now 1790224812691021 us; error 42 us; skew 500 ppm
Sep 24 04:40:12 dist-test-slave-jlnt krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:12.693794 22097 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:40:12.694628 22097 webserver.cc:460] Webserver started at http://127.0.98.62:37199/ using document root <none> and password file <none>
I20260924 04:40:12.695107 22097 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:12.695165 22097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:12.695319 22097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:12.697322 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "2570ea6097da44d580272dc001c5b3be"
format_stamp: "Formatted at 2026-09-24 04:40:12 on dist-test-slave-jlnt"
server_key: "786e618a456a46734b827b868166e2b2"
server_key_iv: "186700d74052f328d685febc1cfa84b5"
server_key_version: "encryptionkey@0"
I20260924 04:40:12.697836 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "2570ea6097da44d580272dc001c5b3be"
format_stamp: "Formatted at 2026-09-24 04:40:12 on dist-test-slave-jlnt"
server_key: "786e618a456a46734b827b868166e2b2"
server_key_iv: "186700d74052f328d685febc1cfa84b5"
server_key_version: "encryptionkey@0"
I20260924 04:40:12.701264 22097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:40:12.703469 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:12.704577 22097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:40:12.704674 22097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "2570ea6097da44d580272dc001c5b3be"
format_stamp: "Formatted at 2026-09-24 04:40:12 on dist-test-slave-jlnt"
server_key: "786e618a456a46734b827b868166e2b2"
server_key_iv: "186700d74052f328d685febc1cfa84b5"
server_key_version: "encryptionkey@0"
I20260924 04:40:12.704751 22097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:12.723644 22097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:12.726681 22097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:12.726822 22097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:12.733388 22097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39035
I20260924 04:40:12.733398 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39035 every 8 connection(s)
I20260924 04:40:12.734246 22097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:12.736779 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:12.739055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 22097
I20260924 04:40:12.739212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:12.739509 392 external_mini_cluster.cc:1428] Setting key 52444ba06f406c5961a851acab4cc898
I20260924 04:40:12.742805 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be: Bootstrap starting.
I20260924 04:40:12.745349 22165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:12.746212 22165 log.cc:824] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:12.748432 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be: No bootstrap required, opened a new log
I20260924 04:40:12.749435 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:12.740895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58451 (local address 127.0.98.62:39035)
0924 04:40:12.741304 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:12.741321 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:12.741363 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:40:12.742165 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:12.742187 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:12.742586 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:12.742946 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:12.743334 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:12.744752 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:12.745732 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:12.746409 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:12.746580 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:12.747501 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:12.747508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:12.747523 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:12.747995 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:12.748058 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:12.748089 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:12.748222 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:12.748407 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 04:40:12.748533 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":100,"threads_started":1}
I20260924 04:40:12.752373 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:12.752597 22165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2570ea6097da44d580272dc001c5b3be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } }
I20260924 04:40:12.752848 22165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:12.752921 22165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2570ea6097da44d580272dc001c5b3be, State: Initialized, Role: FOLLOWER
I20260924 04:40:12.753407 22165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2570ea6097da44d580272dc001c5b3be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } }
I20260924 04:40:12.753566 22165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:12.753647 22165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:12.753813 22165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:12.754840 22165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2570ea6097da44d580272dc001c5b3be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } }
I20260924 04:40:12.755170 22165 leader_election.cc:302] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2570ea6097da44d580272dc001c5b3be; no voters:
I20260924 04:40:12.755460 22165 leader_election.cc:288] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:12.755564 22175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:12.755812 22175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [term 1 LEADER]: Becoming Leader. State: Replica: 2570ea6097da44d580272dc001c5b3be, State: Running, Role: LEADER
I20260924 04:40:12.756145 22175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2570ea6097da44d580272dc001c5b3be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } }
I20260924 04:40:12.756705 22165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:12.757678 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2570ea6097da44d580272dc001c5b3be. Latest consensus state: current_term: 1 leader_uuid: "2570ea6097da44d580272dc001c5b3be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2570ea6097da44d580272dc001c5b3be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } } }
I20260924 04:40:12.757792 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:12.757992 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2570ea6097da44d580272dc001c5b3be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2570ea6097da44d580272dc001c5b3be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } } }
I20260924 04:40:12.758064 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:12.758450 22184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:12.759538 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:12.754612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41135 (local address 127.0.98.62:39035)
0924 04:40:12.754731 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:12.754737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:12.754991 (+ 254us) server_negotiation.cc:407] Connection header received
0924 04:40:12.755131 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:12.755134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:12.755175 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:12.755264 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:12.755550 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:12.756303 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:12.757479 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:12.757870 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:12.757994 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:12.759008 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:12.759014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:12.759021 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:12.759037 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:12.759064 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:12.759079 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:12.759178 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:12.759358 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 04:40:12.759391 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":52}
I20260924 04:40:12.761709 22184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:12.766813 22184 catalog_manager.cc:1347] Generated new cluster ID: cda51cd26c044d618bf716db1ccf5315
I20260924 04:40:12.766920 22184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:12.777738 22184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:12.778656 22184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:12.785113 22184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2570ea6097da44d580272dc001c5b3be: Generated new TSK 0
I20260924 04:40:12.785765 22184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:12.796569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 22097
W20260924 04:40:12.801867 22082 connection.cc:535] client connection to 127.0.98.62:39035 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39035): Connection reset by peer (error 104)
2026-09-24T04:40:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:12 dist-test-slave-jlnt krb5kdc[22196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:12 dist-test-slave-jlnt krb5kdc[22196](info): set up 2 sockets
Sep 24 04:40:12 dist-test-slave-jlnt krb5kdc[22196](info): commencing operation
krb5kdc: starting...
W20260924 04:40:13.829332 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:13 dist-test-slave-jlnt krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:40:13.948279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44341
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:14.056471 22209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:14.056762 22209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:14.056849 22209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:14.060868 22209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:14.060958 22209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:14.060997 22209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:14.061053 22209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:14.061094 22209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:14.066310 22209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:14.067332 22209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:14.068339 22209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:14.076786 22215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:14.076786 22217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:14.076866 22214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:14.077065 22209 server_base.cc:1034] running on GCE node
I20260924 04:40:14.077762 22209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:14.078832 22209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:14.080041 22209 hybrid_clock.cc:648] HybridClock initialized: now 1790224814080017 us; error 41 us; skew 500 ppm
Sep 24 04:40:14 dist-test-slave-jlnt krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:14.082767 22209 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:40:14.083663 22209 webserver.cc:460] Webserver started at http://127.0.98.62:39971/ using document root <none> and password file <none>
I20260924 04:40:14.084149 22209 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:14.084205 22209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:14.084358 22209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:14.086306 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "26a493d5fca7495da264759d20529488"
format_stamp: "Formatted at 2026-09-24 04:40:14 on dist-test-slave-jlnt"
server_key: "891f54fb84bb2f917d3ac4ec39229437"
server_key_iv: "4014942668abd71a2bccc84988818efa"
server_key_version: "encryptionkey@0"
I20260924 04:40:14.086794 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "26a493d5fca7495da264759d20529488"
format_stamp: "Formatted at 2026-09-24 04:40:14 on dist-test-slave-jlnt"
server_key: "891f54fb84bb2f917d3ac4ec39229437"
server_key_iv: "4014942668abd71a2bccc84988818efa"
server_key_version: "encryptionkey@0"
I20260924 04:40:14.090636 22209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:40:14.092877 22224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:14.094081 22209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:40:14.094226 22209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "26a493d5fca7495da264759d20529488"
format_stamp: "Formatted at 2026-09-24 04:40:14 on dist-test-slave-jlnt"
server_key: "891f54fb84bb2f917d3ac4ec39229437"
server_key_iv: "4014942668abd71a2bccc84988818efa"
server_key_version: "encryptionkey@0"
I20260924 04:40:14.094334 22209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:14.098639 22209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:14.101740 22209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:14.101931 22209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:14.108290 22209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44341
I20260924 04:40:14.108349 22276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44341 every 8 connection(s)
I20260924 04:40:14.109208 22209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:14.111716 22277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:14.113619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 22209
I20260924 04:40:14.113772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:14.114041 392 external_mini_cluster.cc:1428] Setting key a3357ed1ae9105bb5710eec61308be1d
I20260924 04:40:14.117659 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488: Bootstrap starting.
I20260924 04:40:14.119270 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:14.115467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45755 (local address 127.0.98.62:44341)
0924 04:40:14.115862 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:14.115877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:14.115910 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:40:14.116710 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:14.116729 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:14.117035 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:14.117379 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:14.118132 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:14.118136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:14.118147 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:14.118156 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:14.118210 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:14.118228 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:14.118330 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:14.118433 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 04:40:14.118591 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":134,"threads_started":1}
I20260924 04:40:14.120060 22277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:14.120759 22277 log.cc:824] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:14.122007 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:14.122758 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488: No bootstrap required, opened a new log
I20260924 04:40:14.125564 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:14.124007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50173 (local address 127.0.98.62:44341)
0924 04:40:14.124113 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:14.124117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:14.124285 (+ 168us) server_negotiation.cc:407] Connection header received
0924 04:40:14.124396 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:14.124401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:14.124432 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:14.124521 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:14.125193 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:14.125196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:14.125199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:14.125209 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:14.125233 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:14.125245 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:14.125337 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:14.125422 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 04:40:14.125457 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:40:14.127055 22277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a493d5fca7495da264759d20529488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44341 } }
I20260924 04:40:14.127249 22277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:14.127326 22277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a493d5fca7495da264759d20529488, State: Initialized, Role: FOLLOWER
I20260924 04:40:14.127722 22277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a493d5fca7495da264759d20529488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44341 } }
I20260924 04:40:14.127866 22277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:14.127934 22277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:14.128046 22277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:14.128860 22277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a493d5fca7495da264759d20529488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44341 } }
I20260924 04:40:14.129238 22277 leader_election.cc:302] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26a493d5fca7495da264759d20529488; no voters:
I20260924 04:40:14.129530 22277 leader_election.cc:288] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:14.129647 22287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:14.129958 22287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [term 1 LEADER]: Becoming Leader. State: Replica: 26a493d5fca7495da264759d20529488, State: Running, Role: LEADER
I20260924 04:40:14.130234 22287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a493d5fca7495da264759d20529488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44341 } }
I20260924 04:40:14.130714 22277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:14.131814 22289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26a493d5fca7495da264759d20529488. Latest consensus state: current_term: 1 leader_uuid: "26a493d5fca7495da264759d20529488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a493d5fca7495da264759d20529488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44341 } } }
I20260924 04:40:14.131950 22289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:14.132287 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26a493d5fca7495da264759d20529488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a493d5fca7495da264759d20529488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44341 } } }
I20260924 04:40:14.132855 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:14.132908 22296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:14.135439 22296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:14.140203 22296 catalog_manager.cc:1347] Generated new cluster ID: c6a69c90311e451c8990718fca39af82
I20260924 04:40:14.140268 22296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:14.154727 22296 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:14.155004 22296 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:14.165093 22296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26a493d5fca7495da264759d20529488: Generated new TSK 0
I20260924 04:40:14.165728 22296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:14.168592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 22209
2026-09-24T04:40:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:14 dist-test-slave-jlnt krb5kdc[22308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:14 dist-test-slave-jlnt krb5kdc[22308](info): set up 2 sockets
Sep 24 04:40:14 dist-test-slave-jlnt krb5kdc[22308](info): commencing operation
krb5kdc: starting...
W20260924 04:40:15.201210 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:15 dist-test-slave-jlnt krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:40:15.320348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:15.427861 22321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:15.428144 22321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:15.428218 22321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:15.432171 22321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:15.432267 22321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:15.432305 22321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:15.432343 22321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:15.432376 22321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:15.437528 22321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:15.438544 22321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:15.439602 22321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:15.448210 22327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:15.448199 22326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:15.448222 22329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:15.448545 22321 server_base.cc:1034] running on GCE node
I20260924 04:40:15.449247 22321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:15.450309 22321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:15.451552 22321 hybrid_clock.cc:648] HybridClock initialized: now 1790224815451535 us; error 34 us; skew 500 ppm
Sep 24 04:40:15 dist-test-slave-jlnt krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:15.465898 22321 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:40:15.467209 22321 webserver.cc:460] Webserver started at http://127.0.98.62:34531/ using document root <none> and password file <none>
I20260924 04:40:15.468057 22321 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:15.468154 22321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:15.468461 22321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:15.470803 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "f2f018d7c11043cba7301c323ffe6533"
format_stamp: "Formatted at 2026-09-24 04:40:15 on dist-test-slave-jlnt"
server_key: "9e93f170ae8212f3c3f09cc4162b27e0"
server_key_iv: "c210dbf4e6c2539f34e55d4ff008ef37"
server_key_version: "encryptionkey@0"
I20260924 04:40:15.471385 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2f018d7c11043cba7301c323ffe6533"
format_stamp: "Formatted at 2026-09-24 04:40:15 on dist-test-slave-jlnt"
server_key: "9e93f170ae8212f3c3f09cc4162b27e0"
server_key_iv: "c210dbf4e6c2539f34e55d4ff008ef37"
server_key_version: "encryptionkey@0"
I20260924 04:40:15.475147 22321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:40:15.477631 22336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:15.478909 22321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:40:15.479065 22321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "f2f018d7c11043cba7301c323ffe6533"
format_stamp: "Formatted at 2026-09-24 04:40:15 on dist-test-slave-jlnt"
server_key: "9e93f170ae8212f3c3f09cc4162b27e0"
server_key_iv: "c210dbf4e6c2539f34e55d4ff008ef37"
server_key_version: "encryptionkey@0"
I20260924 04:40:15.479187 22321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:15.493103 22321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:15.496125 22321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:15.496320 22321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:15.502980 22321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42643
I20260924 04:40:15.502992 22388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42643 every 8 connection(s)
I20260924 04:40:15.503891 22321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:15.506315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 22321
I20260924 04:40:15.506465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:15.506760 392 external_mini_cluster.cc:1428] Setting key b4b9db5a84a838d9e9dab6ee3c010dca
I20260924 04:40:15.506866 22389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:15.513221 22389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533: Bootstrap starting.
Sep 24 04:40:15 dist-test-slave-jlnt krb5kdc[22308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:40:15.515434 22389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:15.516254 22389 log.cc:824] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:15.518226 22389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533: No bootstrap required, opened a new log
I20260924 04:40:15.520370 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:15.508031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50509 (local address 127.0.98.62:42643)
0924 04:40:15.508405 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:15.508420 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:15.508452 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:40:15.509077 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:15.509094 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:15.509476 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:15.509793 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:15.510599 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:15.511567 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:15.512428 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:15.512891 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:15.513051 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:15.515551 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:15.515557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:15.515573 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:15.515617 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:15.517627 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:15.518104 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:15.518110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:15.518116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:15.518233 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:15.518525 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:15.518528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:15.518530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:15.518912 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:15.519115 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:15.519405 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 04:40:15.519687 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":111,"threads_started":1}
I20260924 04:40:15.521957 22389 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f018d7c11043cba7301c323ffe6533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260924 04:40:15.522243 22389 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:15.522349 22389 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f018d7c11043cba7301c323ffe6533, State: Initialized, Role: FOLLOWER
I20260924 04:40:15.522807 22389 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f018d7c11043cba7301c323ffe6533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260924 04:40:15.522971 22389 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:15.523043 22389 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:15.523144 22389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:15.523286 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:15.524225 22389 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f018d7c11043cba7301c323ffe6533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260924 04:40:15.524601 22389 leader_election.cc:302] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2f018d7c11043cba7301c323ffe6533; no voters:
I20260924 04:40:15.524945 22389 leader_election.cc:288] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:15.525063 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:15.525367 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [term 1 LEADER]: Becoming Leader. State: Replica: f2f018d7c11043cba7301c323ffe6533, State: Running, Role: LEADER
I20260924 04:40:15.525701 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f018d7c11043cba7301c323ffe6533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260924 04:40:15.526258 22389 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:15.527055 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2f018d7c11043cba7301c323ffe6533. Latest consensus state: current_term: 1 leader_uuid: "f2f018d7c11043cba7301c323ffe6533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f018d7c11043cba7301c323ffe6533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260924 04:40:15.527199 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:15.527522 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2f018d7c11043cba7301c323ffe6533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f018d7c11043cba7301c323ffe6533" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260924 04:40:15.527581 22408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:15.527630 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:15.530784 22408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:15.535002 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:15.525934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60193 (local address 127.0.98.62:42643)
0924 04:40:15.526047 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:15.526052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:15.526066 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:40:15.526098 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:15.526101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:15.526139 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:15.526218 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:15.527767 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:15.528520 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:15.529645 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:15.530019 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:15.530136 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:15.531734 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:15.531738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:15.531741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:15.531765 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:15.533423 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:15.533824 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:15.533828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:15.533830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:15.533920 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:15.534186 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:15.534191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:15.534193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:15.534365 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:15.534501 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:15.534721 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 04:40:15.534835 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:40:15.537261 22408 catalog_manager.cc:1347] Generated new cluster ID: 1fd0669998e346a8926ccb9c467b13e5
I20260924 04:40:15.537326 22408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:15.547789 22408 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:15.548629 22408 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:15.556875 22408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2f018d7c11043cba7301c323ffe6533: Generated new TSK 0
I20260924 04:40:15.557542 22408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:15.564924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 22321
2026-09-24T04:40:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:15 dist-test-slave-jlnt krb5kdc[22420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:15 dist-test-slave-jlnt krb5kdc[22420](info): set up 2 sockets
Sep 24 04:40:15 dist-test-slave-jlnt krb5kdc[22420](info): commencing operation
krb5kdc: starting...
W20260924 04:40:16.605377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:16 dist-test-slave-jlnt krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:40:16.724067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39603
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:16.828311 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:16.828585 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:16.828697 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:16.832590 22433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:16.832677 22433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:16.832715 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:16.832772 22433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:16.832810 22433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:16.837713 22433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:16.838601 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:16.839516 22433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:16.847162 22438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:16.847187 22441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:16.847378 22433 server_base.cc:1034] running on GCE node
W20260924 04:40:16.847198 22439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:16.847960 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:16.848917 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:16.850073 22433 hybrid_clock.cc:648] HybridClock initialized: now 1790224816850043 us; error 41 us; skew 500 ppm
Sep 24 04:40:16 dist-test-slave-jlnt krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:16.852660 22433 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:40:16.853545 22433 webserver.cc:460] Webserver started at http://127.0.98.62:40569/ using document root <none> and password file <none>
I20260924 04:40:16.854054 22433 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:16.854137 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:16.854351 22433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:16.856191 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "119b0a19d396492fa3da7fe4f638617b"
format_stamp: "Formatted at 2026-09-24 04:40:16 on dist-test-slave-jlnt"
server_key: "06a41024a62fa3185b351c4fab415eec"
server_key_iv: "85e2c20c117c70ed24c4e0fec7f0c9d7"
server_key_version: "encryptionkey@0"
I20260924 04:40:16.856740 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "119b0a19d396492fa3da7fe4f638617b"
format_stamp: "Formatted at 2026-09-24 04:40:16 on dist-test-slave-jlnt"
server_key: "06a41024a62fa3185b351c4fab415eec"
server_key_iv: "85e2c20c117c70ed24c4e0fec7f0c9d7"
server_key_version: "encryptionkey@0"
I20260924 04:40:16.860116 22433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:40:16.862354 22448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:16.863462 22433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:40:16.863590 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "119b0a19d396492fa3da7fe4f638617b"
format_stamp: "Formatted at 2026-09-24 04:40:16 on dist-test-slave-jlnt"
server_key: "06a41024a62fa3185b351c4fab415eec"
server_key_iv: "85e2c20c117c70ed24c4e0fec7f0c9d7"
server_key_version: "encryptionkey@0"
I20260924 04:40:16.863691 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:16.872243 22433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:16.874770 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:16.874953 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:16.881453 22433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39603
I20260924 04:40:16.881481 22500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39603 every 8 connection(s)
I20260924 04:40:16.882288 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:16.884814 22501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:16.888734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 22433
I20260924 04:40:16.888867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:16.889169 392 external_mini_cluster.cc:1428] Setting key 2c8e3a0e8c058932711f3665816b74c6
I20260924 04:40:16.890385 22501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b: Bootstrap starting.
I20260924 04:40:16.892539 22501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:16.893260 22501 log.cc:824] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:16.895152 22501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b: No bootstrap required, opened a new log
I20260924 04:40:16.897529 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:16.890414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57197 (local address 127.0.98.62:39603)
0924 04:40:16.890828 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:16.890860 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:16.890893 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:40:16.891478 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:16.891493 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:16.891798 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:16.892061 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:16.892372 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:16.893460 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:16.894591 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:16.895032 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:16.895146 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:16.895930 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:16.895935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:16.895946 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:16.896305 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:16.896352 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:16.896373 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:16.896445 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:16.896604 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 04:40:16.896810 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
I20260924 04:40:16.898880 22501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119b0a19d396492fa3da7fe4f638617b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39603 } }
I20260924 04:40:16.899123 22501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:16.899242 22501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119b0a19d396492fa3da7fe4f638617b, State: Initialized, Role: FOLLOWER
I20260924 04:40:16.899689 22501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119b0a19d396492fa3da7fe4f638617b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39603 } }
I20260924 04:40:16.899809 22501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:16.899890 22501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:16.899997 22501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:16.900276 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:16.900866 22501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119b0a19d396492fa3da7fe4f638617b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39603 } }
I20260924 04:40:16.901227 22501 leader_election.cc:302] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 119b0a19d396492fa3da7fe4f638617b; no voters:
I20260924 04:40:16.901513 22501 leader_election.cc:288] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:16.901623 22507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:16.901930 22507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [term 1 LEADER]: Becoming Leader. State: Replica: 119b0a19d396492fa3da7fe4f638617b, State: Running, Role: LEADER
I20260924 04:40:16.902258 22507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119b0a19d396492fa3da7fe4f638617b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39603 } }
I20260924 04:40:16.902673 22501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:16.903923 22512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 119b0a19d396492fa3da7fe4f638617b. Latest consensus state: current_term: 1 leader_uuid: "119b0a19d396492fa3da7fe4f638617b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119b0a19d396492fa3da7fe4f638617b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39603 } } }
I20260924 04:40:16.904053 22512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:16.904501 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "119b0a19d396492fa3da7fe4f638617b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119b0a19d396492fa3da7fe4f638617b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39603 } } }
I20260924 04:40:16.904634 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:16.907255 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:16.902385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53307 (local address 127.0.98.62:39603)
0924 04:40:16.902474 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:16.902478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:16.902847 (+ 369us) server_negotiation.cc:407] Connection header received
0924 04:40:16.902952 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:16.902963 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:16.903123 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:16.903204 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:16.903493 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:16.904294 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:16.905504 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:16.905743 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:16.905898 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:16.906642 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:16.906646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:16.906648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:16.906659 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:16.906684 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:16.906698 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:16.906759 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:16.907131 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 04:40:16.907169 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 04:40:16.907335 22526 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:16.907431 22526 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:40:16.907531 22522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:16.908166 22522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:16.912997 22522 catalog_manager.cc:1347] Generated new cluster ID: e09af42e2d114f0383f2ef7c0b1ace22
I20260924 04:40:16.913059 22522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:16.920542 22522 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:16.921627 22522 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:16.937738 22522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 119b0a19d396492fa3da7fe4f638617b: Generated new TSK 0
I20260924 04:40:16.938436 22522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:16.943663 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 22433
2026-09-24T04:40:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:16 dist-test-slave-jlnt krb5kdc[22532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:16 dist-test-slave-jlnt krb5kdc[22532](info): set up 2 sockets
Sep 24 04:40:16 dist-test-slave-jlnt krb5kdc[22532](info): commencing operation
krb5kdc: starting...
W20260924 04:40:17.985460 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:18 dist-test-slave-jlnt krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:40:18.103461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:18.210177 22545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:18.210423 22545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:18.210461 22545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:18.214438 22545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:18.214493 22545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:18.214512 22545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:18.214529 22545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:18.214582 22545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:18.219722 22545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:18.220683 22545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:18.221658 22545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:18.229660 22551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:18.229626 22550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:18.229677 22545 server_base.cc:1034] running on GCE node
W20260924 04:40:18.229625 22553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:18.230396 22545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:18.231384 22545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:18.232545 22545 hybrid_clock.cc:648] HybridClock initialized: now 1790224818232532 us; error 33 us; skew 500 ppm
Sep 24 04:40:18 dist-test-slave-jlnt krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:18.243355 22545 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:40:18.244280 22545 webserver.cc:460] Webserver started at http://127.0.98.62:41251/ using document root <none> and password file <none>
I20260924 04:40:18.244822 22545 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:18.244910 22545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:18.245186 22545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:18.247249 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "cd77de8f3ec741199d5d4a0418829510"
format_stamp: "Formatted at 2026-09-24 04:40:18 on dist-test-slave-jlnt"
server_key: "895fe41b3391e7bca471f9f23a6409c8"
server_key_iv: "fb929473087e90135c3b1be424cd71f0"
server_key_version: "encryptionkey@0"
I20260924 04:40:18.247813 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd77de8f3ec741199d5d4a0418829510"
format_stamp: "Formatted at 2026-09-24 04:40:18 on dist-test-slave-jlnt"
server_key: "895fe41b3391e7bca471f9f23a6409c8"
server_key_iv: "fb929473087e90135c3b1be424cd71f0"
server_key_version: "encryptionkey@0"
I20260924 04:40:18.251447 22545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:40:18.253765 22560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:18.255026 22545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:40:18.255173 22545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "cd77de8f3ec741199d5d4a0418829510"
format_stamp: "Formatted at 2026-09-24 04:40:18 on dist-test-slave-jlnt"
server_key: "895fe41b3391e7bca471f9f23a6409c8"
server_key_iv: "fb929473087e90135c3b1be424cd71f0"
server_key_version: "encryptionkey@0"
I20260924 04:40:18.255280 22545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:18.259601 22545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:18.263775 22545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:18.263955 22545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:18.270781 22545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39487
I20260924 04:40:18.270812 22612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39487 every 8 connection(s)
I20260924 04:40:18.271632 22545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:18.274160 22613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:18.278672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 22545
I20260924 04:40:18.278815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:18.279073 392 external_mini_cluster.cc:1428] Setting key a375ce3119bbcd968e5bd3d8104e23e2
I20260924 04:40:18.279505 22613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510: Bootstrap starting.
I20260924 04:40:18.281829 22613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:18.282686 22613 log.cc:824] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:18.284590 22613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510: No bootstrap required, opened a new log
I20260924 04:40:18.284603 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:18.280322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46895 (local address 127.0.98.62:39487)
0924 04:40:18.280746 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:18.280762 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:18.280796 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:40:18.281453 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:18.281470 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:18.281736 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:18.282094 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:18.283479 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:18.283485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:18.283500 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:18.283512 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:18.283578 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:18.283597 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:18.283711 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:18.283825 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:40:18.283976 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":132,"threads_started":1}
I20260924 04:40:18.287415 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:18.288473 22613 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd77de8f3ec741199d5d4a0418829510" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } }
I20260924 04:40:18.288756 22613 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:40:18.288826 22613 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd77de8f3ec741199d5d4a0418829510, State: Initialized, Role: FOLLOWER
I20260924 04:40:18.289305 22613 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd77de8f3ec741199d5d4a0418829510" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } }
I20260924 04:40:18.289471 22613 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:40:18.289606 22613 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:40:18.289731 22613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:18.290548 22613 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd77de8f3ec741199d5d4a0418829510" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } }
I20260924 04:40:18.290877 22613 leader_election.cc:302] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd77de8f3ec741199d5d4a0418829510; no voters:
I20260924 04:40:18.291153 22613 leader_election.cc:288] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:40:18.291261 22623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:18.291460 22623 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [term 1 LEADER]: Becoming Leader. State: Replica: cd77de8f3ec741199d5d4a0418829510, State: Running, Role: LEADER
I20260924 04:40:18.291823 22623 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd77de8f3ec741199d5d4a0418829510" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } }
I20260924 04:40:18.291935 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:18.289450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52793 (local address 127.0.98.62:39487)
0924 04:40:18.289571 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:18.289575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:18.289735 (+ 160us) server_negotiation.cc:407] Connection header received
0924 04:40:18.289872 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:18.289876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:18.289905 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:18.289994 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:18.291438 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:18.291444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:18.291448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:40:18.291460 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:18.291485 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:40:18.291499 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:18.291561 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:18.291731 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 04:40:18.291766 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:40:18.292423 22613 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:18.293203 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd77de8f3ec741199d5d4a0418829510. Latest consensus state: current_term: 1 leader_uuid: "cd77de8f3ec741199d5d4a0418829510" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd77de8f3ec741199d5d4a0418829510" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } } }
I20260924 04:40:18.293311 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:18.293799 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd77de8f3ec741199d5d4a0418829510" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd77de8f3ec741199d5d4a0418829510" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } } }
I20260924 04:40:18.294092 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:18.294378 22634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:18.296864 22634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:18.302027 22634 catalog_manager.cc:1347] Generated new cluster ID: 29cd787eed3043ae93a843d18aefed2e
I20260924 04:40:18.302085 22634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:18.327428 22634 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:18.327765 22634 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:18.335083 22634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd77de8f3ec741199d5d4a0418829510: Generated new TSK 0
I20260924 04:40:18.335835 22634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:40:18.341046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 22545
2026-09-24T04:40:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1397 ms)
[----------] 18 tests from AuthTokenIssuingTest (24957 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:18 dist-test-slave-jlnt krb5kdc[22644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:18 dist-test-slave-jlnt krb5kdc[22644](info): set up 2 sockets
Sep 24 04:40:18 dist-test-slave-jlnt krb5kdc[22644](info): commencing operation
krb5kdc: starting...
W20260924 04:40:19.373422 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:19 dist-test-slave-jlnt krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:40:19.491822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35365
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35365,127.0.98.61:45203,127.0.98.60:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:19.598702 22657 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:40:19.598958 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:19.598981 22657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:19.599018 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:19.602871 22657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:19.602916 22657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:19.602936 22657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:19.602952 22657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:19.602967 22657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:19.607939 22657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35365,127.0.98.61:45203,127.0.98.60:45339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:19.608846 22657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:19.609778 22657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:19.617864 22662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:19.617843 22665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:19.617864 22663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:19.618161 22657 server_base.cc:1034] running on GCE node
I20260924 04:40:19.618809 22657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:19.619836 22657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:19.620994 22657 hybrid_clock.cc:648] HybridClock initialized: now 1790224819620983 us; error 26 us; skew 500 ppm
Sep 24 04:40:19 dist-test-slave-jlnt krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:19.624533 22657 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:40:19.625479 22657 webserver.cc:460] Webserver started at http://127.0.98.62:41761/ using document root <none> and password file <none>
I20260924 04:40:19.626010 22657 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:19.626084 22657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:19.626292 22657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:19.628263 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "d3bb3ef95c8c442e98e665bc4eac0242"
format_stamp: "Formatted at 2026-09-24 04:40:19 on dist-test-slave-jlnt"
server_key: "e060e7c694c47ebf2ddca6e60529650f"
server_key_iv: "7201fd1f3d4268a756770e4644851b08"
server_key_version: "encryptionkey@0"
I20260924 04:40:19.628813 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3bb3ef95c8c442e98e665bc4eac0242"
format_stamp: "Formatted at 2026-09-24 04:40:19 on dist-test-slave-jlnt"
server_key: "e060e7c694c47ebf2ddca6e60529650f"
server_key_iv: "7201fd1f3d4268a756770e4644851b08"
server_key_version: "encryptionkey@0"
I20260924 04:40:19.632349 22657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:40:19.634740 22672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:19.635932 22657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:40:19.636081 22657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "d3bb3ef95c8c442e98e665bc4eac0242"
format_stamp: "Formatted at 2026-09-24 04:40:19 on dist-test-slave-jlnt"
server_key: "e060e7c694c47ebf2ddca6e60529650f"
server_key_iv: "7201fd1f3d4268a756770e4644851b08"
server_key_version: "encryptionkey@0"
I20260924 04:40:19.636188 22657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:19.645984 22657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:19.651176 22657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:19.651373 22657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:19.658023 22657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35365
I20260924 04:40:19.658305 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35365 every 8 connection(s)
I20260924 04:40:19.658941 22657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:19.661677 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:19.665263 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:19.667106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 22657
I20260924 04:40:19.667237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:19.667506 392 external_mini_cluster.cc:1428] Setting key ca4acdecbeee549507f68ccc2f034f25
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Sep 24 04:40:19 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 04:40:19.682802 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.667426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.0.98.62:35365)
0924 04:40:19.667989 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:19.668002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:19.668046 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:40:19.669078 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:19.669082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:19.669334 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:19.669482 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:19.670565 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.671465 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.673066 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.673492 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.673609 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:19.676716 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:19.676729 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:19.676738 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:19.676756 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:19.679722 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:19.680275 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:19.680283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:19.680286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:19.680428 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:19.680727 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:19.680732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:19.680735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:19.681265 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:19.681359 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:19.681826 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 04:40:19.681981 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"spinlock_wait_cycles":88960,"thread_start_us":181,"threads_started":1}
I20260924 04:40:19.682806 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.667425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35365 (local address 127.0.0.1:56282)
0924 04:40:19.667873 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.667922 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:19.668722 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:19.669536 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:19.669543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:19.670026 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:19.670407 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:19.670428 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.671600 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.671603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:19.672914 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:19.672929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.673654 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.673660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:19.673798 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:19.674935 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:19.674958 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:19.676574 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:19.679867 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:19.679880 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:19.679900 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:19.680167 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:19.680530 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:19.680532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:19.680533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:19.680637 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:19.681390 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:19.681400 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:19.681575 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 04:40:19.681759 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 04:40:19.682069 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":145,"threads_started":1}
I20260924 04:40:19.685276 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:19.686110 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.685769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:58084)
0924 04:40:19.685923 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.686030 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 04:40:19.686403 22676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
I20260924 04:40:19.687186 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.686994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:58096)
0924 04:40:19.687098 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.687140 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:40:19.687322 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45203: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:40:19.695808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45203
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35365,127.0.98.61:45203,127.0.98.60:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:40:19.736570 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attempt: 1
I20260924 04:40:19.737440 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.737159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:58110)
0924 04:40:19.737305 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.737367 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:40:19.738214 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.738025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:58118)
0924 04:40:19.738123 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.738164 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:40:19.738415 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45203: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
W20260924 04:40:19.804026 22732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:40:19.804275 22732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:19.804297 22732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:19.804334 22732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:40:19.806595 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attempt: 2
I20260924 04:40:19.807402 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.807122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:58124)
0924 04:40:19.807271 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.807331 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 04:40:19.808135 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.807948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:58126)
0924 04:40:19.808045 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.808088 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:40:19.808317 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45203: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45203: connect: Connection refused (error 111)
W20260924 04:40:19.808949 22732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:19.809000 22732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:19.809039 22732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:19.809075 22732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:19.809123 22732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:19.814599 22732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35365,127.0.98.61:45203,127.0.98.60:45339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:19.815560 22732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:19.816504 22732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:19.824563 22738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:19.824555 22740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:19.824708 22737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:19.824975 22732 server_base.cc:1034] running on GCE node
I20260924 04:40:19.825601 22732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:19.826671 22732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:19.827831 22732 hybrid_clock.cc:648] HybridClock initialized: now 1790224819827799 us; error 47 us; skew 500 ppm
Sep 24 04:40:19 dist-test-slave-jlnt krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:19.831130 22732 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:40:19.832008 22732 webserver.cc:460] Webserver started at http://127.0.98.61:45481/ using document root <none> and password file <none>
I20260924 04:40:19.832557 22732 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:19.832646 22732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:19.832868 22732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:19.834897 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/data/instance:
uuid: "8b6292b3f31940ca86e72fcb7c569e13"
format_stamp: "Formatted at 2026-09-24 04:40:19 on dist-test-slave-jlnt"
server_key: "a467fdd2417d5b077c07abedc41ffe32"
server_key_iv: "c62bec572e1d6b82eb6d58b894d56ee1"
server_key_version: "encryptionkey@0"
I20260924 04:40:19.835467 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/wal/instance:
uuid: "8b6292b3f31940ca86e72fcb7c569e13"
format_stamp: "Formatted at 2026-09-24 04:40:19 on dist-test-slave-jlnt"
server_key: "a467fdd2417d5b077c07abedc41ffe32"
server_key_iv: "c62bec572e1d6b82eb6d58b894d56ee1"
server_key_version: "encryptionkey@0"
I20260924 04:40:19.839015 22732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:40:19.841516 22747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:19.842674 22732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:40:19.842808 22732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/wal
uuid: "8b6292b3f31940ca86e72fcb7c569e13"
format_stamp: "Formatted at 2026-09-24 04:40:19 on dist-test-slave-jlnt"
server_key: "a467fdd2417d5b077c07abedc41ffe32"
server_key_iv: "c62bec572e1d6b82eb6d58b894d56ee1"
server_key_version: "encryptionkey@0"
I20260924 04:40:19.842912 22732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:19.860466 22732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:19.864394 22732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:19.864610 22732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:19.871467 22732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45203
I20260924 04:40:19.871739 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45203 every 8 connection(s)
I20260924 04:40:19.872396 22732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:40:19.874961 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:19.878454 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:19.881369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 22732
I20260924 04:40:19.881491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-1/wal/instance
I20260924 04:40:19.881778 392 external_mini_cluster.cc:1428] Setting key 8e4dd7f86b57712d562d81c7ee35d418
Sep 24 04:40:19 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 04:40:19.894785 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.880201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.0.98.62:35365)
0924 04:40:19.880334 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:19.880338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:19.881095 (+ 757us) server_negotiation.cc:407] Connection header received
0924 04:40:19.882197 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:19.882201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:19.882265 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:19.882369 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:19.883433 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.883955 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.885729 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.885968 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.886069 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:19.889444 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:19.889448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:19.889450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:19.889473 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:19.892380 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:19.892991 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:19.892994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:19.892995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:19.893088 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:19.893502 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:19.893505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:19.893506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:19.893680 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:19.893754 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:19.894425 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 04:40:19.894571 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:40:19.895454 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.880538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35365 (local address 127.0.0.1:56298)
0924 04:40:19.880956 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.880995 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:19.881991 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:19.882526 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:19.882532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:19.882848 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:19.883284 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:19.883296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.884099 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.884106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:19.885580 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:19.885595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.886141 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.886146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:19.886376 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:19.887586 (+ 1210us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:19.887614 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:19.889273 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:19.892513 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:19.892519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:19.892533 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:19.892853 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:19.893207 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:19.893213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:19.893218 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:19.893404 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:19.893779 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:19.893788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:19.894113 (+ 325us) client_negotiation.cc:769] Sending connection context
0924 04:40:19.894334 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 04:40:19.894601 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260924 04:40:19.896808 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:19.899480 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attempt: 3
Sep 24 04:40:19 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:40:19 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:40:19.909435 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.897444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.0.98.61:45203)
0924 04:40:19.897759 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:19.897768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:19.897784 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:40:19.897814 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:19.897817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:19.897952 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:19.898089 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:19.898408 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.899156 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.900390 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.900810 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.900897 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:19.903856 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:19.903864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:19.903877 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:19.903903 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:19.907258 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:19.907753 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:19.907761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:19.907766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:19.907869 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:19.908176 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:19.908178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:19.908180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:19.908447 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:19.908609 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:19.908992 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 04:40:19.909225 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":228,"thread_start_us":133,"threads_started":1}
I20260924 04:40:19.909237 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.897338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:58128)
0924 04:40:19.897488 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.897499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:19.897559 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:19.898069 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:19.898071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:19.898242 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:19.898308 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:19.898313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.899272 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.899275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:19.900273 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:19.900280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.900899 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.900902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:19.901028 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:19.902279 (+ 1251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:19.902291 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:19.903733 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:19.907368 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:19.907372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:19.907374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:19.907639 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:19.907970 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:19.907975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:19.907979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:19.908087 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:19.908653 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:19.908655 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:19.908772 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 04:40:19.908880 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:40:19.909022 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":1}
I20260924 04:40:19.909740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45339
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35365,127.0.98.61:45203,127.0.98.60:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:40:19.912184 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.899853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:58138)
0924 04:40:19.899954 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.899967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:19.900047 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:19.900463 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:19.900466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:19.900629 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:19.900705 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:19.900709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.902096 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.902099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:19.902899 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:19.902913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.903434 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.903441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:19.903545 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:19.904821 (+ 1276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:19.904837 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:19.906227 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:19.910458 (+ 4231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:19.910462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:19.910464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:19.910669 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:19.911013 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:19.911017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:19.911020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:19.911117 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:19.911595 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:19.911600 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:19.911724 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 04:40:19.911863 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 04:40:19.912008 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 04:40:19.912691 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:19.912747 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.899932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.0.98.61:45203)
0924 04:40:19.900233 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:19.900241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:19.900253 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:40:19.900282 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:19.900286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:19.900329 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:19.900445 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:19.900813 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.901992 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.903031 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:19.903321 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:19.903398 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:19.906410 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:19.906413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:19.906415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:19.906435 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:19.910343 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:19.910801 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:19.910807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:19.910811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:19.910907 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:19.911233 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:19.911239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:19.911241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:19.911464 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:19.912391 (+ 927us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:19.912465 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:40:19.912582 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260924 04:40:19.913507 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:19.913641 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.913115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35030)
0924 04:40:19.913411 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.913494 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
W20260924 04:40:19.913895 22750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
I20260924 04:40:19.914146 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.913893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35034)
0924 04:40:19.914013 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.914066 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:40:19.914983 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.914775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35044)
0924 04:40:19.914887 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.914930 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 04:40:19.915122 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
I20260924 04:40:19.915310 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.914937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35056)
0924 04:40:19.915176 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.915235 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260924 04:40:19.915436 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
I20260924 04:40:19.946295 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } attempt: 1
I20260924 04:40:19.947135 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.946860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35072)
0924 04:40:19.946998 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.947058 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 04:40:19.947817 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.947607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35084)
0924 04:40:19.947715 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.947765 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:40:19.947978 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
I20260924 04:40:19.964694 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } attempt: 1
I20260924 04:40:19.965581 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.965258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35094)
0924 04:40:19.965409 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.965479 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:40:19.966396 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:19.966162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35098)
0924 04:40:19.966278 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:19.966333 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:40:19.966554 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
W20260924 04:40:20.020033 22807 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:40:20.020354 22807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:20.020426 22807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:20.020503 22807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:40:20.023171 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } attempt: 2
I20260924 04:40:20.023917 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.023637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35108)
0924 04:40:20.023776 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:20.023841 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:40:20.024678 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.024435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35124)
0924 04:40:20.024578 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:20.024630 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 04:40:20.024829 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
W20260924 04:40:20.025043 22807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:20.025084 22807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:20.025103 22807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:20.025125 22807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:20.025195 22807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:20.030347 22807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35365,127.0.98.61:45203,127.0.98.60:45339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:20.031246 22807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:20.032151 22807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:40:20.034749 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } attempt: 2
I20260924 04:40:20.035601 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.035270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35134)
0924 04:40:20.035455 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:20.035525 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 04:40:20.036410 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.036149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35140)
0924 04:40:20.036267 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:20.036330 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 04:40:20.036579 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45339: connect: Connection refused (error 111)
W20260924 04:40:20.040666 22814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:20.040673 22816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:20.040810 22807 server_base.cc:1034] running on GCE node
W20260924 04:40:20.040673 22813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:20.041481 22807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:20.042481 22807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:20.043666 22807 hybrid_clock.cc:648] HybridClock initialized: now 1790224820043648 us; error 35 us; skew 500 ppm
Sep 24 04:40:20 dist-test-slave-jlnt krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:20.047336 22807 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:40:20.048149 22807 webserver.cc:460] Webserver started at http://127.0.98.60:44353/ using document root <none> and password file <none>
I20260924 04:40:20.048693 22807 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:20.048781 22807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:20.048985 22807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:20.050951 22807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/data/instance:
uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe"
format_stamp: "Formatted at 2026-09-24 04:40:20 on dist-test-slave-jlnt"
server_key: "004abd7f6e662b17d34e64ccdc0d3fae"
server_key_iv: "6431cf674e8bdca4514f874db62632c4"
server_key_version: "encryptionkey@0"
I20260924 04:40:20.051498 22807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/wal/instance:
uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe"
format_stamp: "Formatted at 2026-09-24 04:40:20 on dist-test-slave-jlnt"
server_key: "004abd7f6e662b17d34e64ccdc0d3fae"
server_key_iv: "6431cf674e8bdca4514f874db62632c4"
server_key_version: "encryptionkey@0"
I20260924 04:40:20.054934 22807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:40:20.057226 22823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:20.058338 22807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:40:20.058482 22807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/wal
uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe"
format_stamp: "Formatted at 2026-09-24 04:40:20 on dist-test-slave-jlnt"
server_key: "004abd7f6e662b17d34e64ccdc0d3fae"
server_key_iv: "6431cf674e8bdca4514f874db62632c4"
server_key_version: "encryptionkey@0"
I20260924 04:40:20.058584 22807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:20.097651 22807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:20.103070 22807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:20.103305 22807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:20.110538 22807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45339
I20260924 04:40:20.110809 22875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45339 every 8 connection(s)
I20260924 04:40:20.111493 22807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:40:20.114192 22876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:20.116595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 22807
I20260924 04:40:20.116734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790224545298817-392-0/minicluster-data/master-2/wal/instance
I20260924 04:40:20.117022 392 external_mini_cluster.cc:1428] Setting key 2a609755444c013df9644ee6f6271584
I20260924 04:40:20.118330 22876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:20.123991 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } attempt: 3
Sep 24 04:40:20 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:40:20 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:40:20.127777 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } attempt: 3
Sep 24 04:40:20 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:40:20.135134 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.119962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.0.98.62:35365)
0924 04:40:20.120098 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.120104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.120409 (+ 305us) server_negotiation.cc:407] Connection header received
0924 04:40:20.120624 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.120626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.120669 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.120736 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.121592 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.122329 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.123742 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.124085 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.124185 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.127730 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.127736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.127742 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.127773 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.131738 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.132627 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.132631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.132633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.132724 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.133447 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.133454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.133459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.134186 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.134301 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.134819 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.134934 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 04:40:20 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:40:20.136760 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.118287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.0.98.60:45339)
0924 04:40:20.118690 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.118709 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.118751 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:40:20.119391 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.119407 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.119807 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.120134 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.121222 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.122543 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.123742 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.124370 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.124483 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.128830 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.128840 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.128855 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.128882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.132749 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.133555 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.133560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.133562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.133646 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.133964 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.133968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.133969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.134465 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.135255 (+ 790us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.135363 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.135657 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":410,"server-negotiator.queue_time_us":248,"thread_start_us":121,"threads_started":1}
I20260924 04:40:20.136761 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.119883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35365 (local address 127.0.0.1:56312)
0924 04:40:20.120271 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:20.120327 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:20.120503 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:20.120819 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:20.120825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:20.121290 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:20.121452 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.121461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.122436 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.122439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.123634 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.123644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.124262 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.124270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.124467 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.125897 (+ 1430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:20.125912 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:20.127592 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:20.131852 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.131860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.131871 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.132515 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.132843 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.132846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.132848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.133025 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.134364 (+ 1339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:20.134371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:20.134533 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 04:40:20.135340 (+ 807us) client_negotiation.cc:240] Negotiation successful
0924 04:40:20.135750 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":113,"thread_start_us":95,"threads_started":1}
I20260924 04:40:20.138463 22876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:20.142671 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.124407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35154)
0924 04:40:20.124589 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:20.124604 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:20.124680 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:20.126194 (+ 1514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:20.126197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:20.126364 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:20.126482 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.126492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.128622 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.128627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.129540 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.129552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.130181 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.130187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.130302 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.132299 (+ 1997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:20.132313 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:20.133754 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:20.140754 (+ 7000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.140757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.140759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.141295 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.141613 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.141616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.141618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.141702 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.142214 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:20.142218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:20.142304 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 04:40:20.142386 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:40:20.142524 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 04:40:20.142671 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.128220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35164)
0924 04:40:20.128387 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:20.128399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:20.128470 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:20.129697 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:20.129701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:20.129912 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:20.130062 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.130078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.131329 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.131336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.132214 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.132226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.132880 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.132886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.132969 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.134124 (+ 1155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:20.134138 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:20.135668 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:20.140756 (+ 5088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.140759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.140761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.140993 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.141615 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.141618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.141620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.141702 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.142214 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:20.142217 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:20.142304 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 04:40:20.142396 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:40:20.142515 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 04:40:20.143304 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.124449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.0.98.60:45339)
0924 04:40:20.124694 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.124705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.124719 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:40:20.125991 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.125997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.126054 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.126151 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.127729 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.128449 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.129678 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.130057 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.130179 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.133899 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.133907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.133912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.133942 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.140586 (+ 6644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.141391 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.141397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.141402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.141490 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.141812 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.141815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.141817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.142100 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.142917 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.142998 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.143128 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":190,"spinlock_wait_cycles":8704,"thread_start_us":83,"threads_started":1}
I20260924 04:40:20.143345 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.128283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.0.98.60:45339)
0924 04:40:20.128773 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.128783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.128875 (+ 92us) server_negotiation.cc:407] Connection header received
0924 04:40:20.128906 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.128911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.128975 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.129677 (+ 702us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.130317 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.131183 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.132347 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.132725 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.133233 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.135833 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.135837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.135839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.135866 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.140641 (+ 4775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.141445 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.141449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.141453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.141513 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.141812 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.141815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.141817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.142100 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.142917 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.143013 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.143203 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":216,"server-negotiator.queue_time_us":383,"spinlock_wait_cycles":2560,"thread_start_us":142,"threads_started":1}
Sep 24 04:40:20 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:40:20.146566 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13: Bootstrap starting.
I20260924 04:40:20.146566 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242: Bootstrap starting.
Sep 24 04:40:20 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:40:20.149291 22725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:20.149291 22800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:20.150233 22725 log.cc:824] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:20.150233 22800 log.cc:824] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:20.152310 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13: No bootstrap required, opened a new log
I20260924 04:40:20.152310 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242: No bootstrap required, opened a new log
I20260924 04:40:20.152601 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.139248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.0.98.61:45203)
0924 04:40:20.139360 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.139364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.139681 (+ 317us) server_negotiation.cc:407] Connection header received
0924 04:40:20.139842 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.139847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.139894 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.139945 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.140595 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.141418 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.142451 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.142825 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.142911 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.146125 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.146130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.146132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.146168 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.150662 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.151197 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.151203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.151205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.151288 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.151615 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.151625 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.151629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.151890 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.151959 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.152336 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.152456 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:40:20.153083 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.139158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:58142)
0924 04:40:20.139617 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:20.139631 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:20.139738 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:20.140101 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:20.140111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:20.140306 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:20.140440 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.140450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.141543 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.141546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.142327 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.142336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.143060 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.143064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.143159 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.144331 (+ 1172us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:20.144345 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:20.145938 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:20.150805 (+ 4867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.150807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.150809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.151076 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.151403 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.151406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.151408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.151504 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.152045 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:20.152050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:20.152149 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 04:40:20.152768 (+ 619us) client_negotiation.cc:240] Negotiation successful
0924 04:40:20.152912 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372}
I20260924 04:40:20.153815 22876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:20.153905 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.139912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.0.98.62:35365)
0924 04:40:20.140053 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.140059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.140076 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:40:20.140117 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.140119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.140161 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.140243 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.140979 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.141731 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.143220 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.143621 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.143709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.148061 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.148066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.148070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.148102 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.151437 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.151938 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.151942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.151943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.152026 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.152331 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.152334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.152336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.152537 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.152718 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.153651 (+ 933us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.153766 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:40:20.156296 22800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } }
I20260924 04:40:20.156296 22725 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } }
I20260924 04:40:20.156522 22725 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bb3ef95c8c442e98e665bc4eac0242, State: Initialized, Role: FOLLOWER
I20260924 04:40:20.156522 22800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6292b3f31940ca86e72fcb7c569e13, State: Initialized, Role: FOLLOWER
I20260924 04:40:20.156875 22725 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } }
I20260924 04:40:20.156875 22800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } }
I20260924 04:40:20.158222 22800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:20.158222 22725 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 04:40:20 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:40:20 dist-test-slave-jlnt krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260924 04:40:20.163993 22907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:20.164078 22907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:40:20.164306 22887 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.164403 22887 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:20.164614 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.164727 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 04:40:20.167778 22909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:20.167861 22909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:40:20.167584 22732 master_runner.cc:186] Error getting master registration for 127.0.98.60:45339: OK, instance_id { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" instance_seqno: 1790224820108498 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:40:20.171059 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.154503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.98.60:45339)
0924 04:40:20.154614 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.154620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.154752 (+ 132us) server_negotiation.cc:407] Connection header received
0924 04:40:20.154889 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.154892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.154928 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.154984 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.155695 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.156291 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.157684 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.158025 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.158090 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.161200 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.161207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.161216 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.161245 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.167679 (+ 6434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.168688 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.168691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.168695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.168767 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.170207 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.170211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.170213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.170412 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.170480 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.170750 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.170865 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:40:20.171149 22732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:40:20.171509 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.154273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45339 (local address 127.0.0.1:35172)
0924 04:40:20.154700 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:20.154712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:20.154804 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:20.155112 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:20.155325 (+ 213us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:20.155493 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:20.155579 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.155589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.156717 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.156722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.157582 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:20.157592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.158217 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.158220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:20.158297 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.159448 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:20.159467 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:20.161036 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:20.168153 (+ 7117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.168156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.168158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.168385 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.169086 (+ 701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:20.169089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:20.169091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:20.169200 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:20.170540 (+ 1340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:20.170543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:20.170613 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 04:40:20.171227 (+ 614us) client_negotiation.cc:240] Negotiation successful
0924 04:40:20.171363 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":92,"threads_started":1}
I20260924 04:40:20.172194 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.155698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.0.98.61:45203)
0924 04:40:20.156411 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.156414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.156423 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:40:20.156447 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.156449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.156482 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.156582 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.157530 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.158250 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.159528 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.159875 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.159971 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.163974 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.163977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.163980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.164008 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.167102 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.168503 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.168507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.168508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.168587 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.169303 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.169307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.169309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.169691 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.169881 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.171906 (+ 2025us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.172042 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653}
I20260924 04:40:20.174465 22876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe: Bootstrap starting.
I20260924 04:40:20.175801 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:20.177042 22699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d3bb3ef95c8c442e98e665bc4eac0242"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56316
I20260924 04:40:20.177250 22699 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:40:20.177297 22876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:20.177340 22699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:20.174412 22657 master_runner.cc:186] Error getting master registration for 127.0.98.60:45339: OK, instance_id { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" instance_seqno: 1790224820108498 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:40:20.177852 22657 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:40:20.178130 22876 log.cc:824] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:20.178562 22699 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } }
I20260924 04:40:20.179553 22699 leader_election.cc:288] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [CANDIDATE]: Term 1 election: Requested vote from peers 8b6292b3f31940ca86e72fcb7c569e13 (127.0.98.61:45203), 7dfb578c4ef14fc3b0cd69d15a2de4fe (127.0.98.60:45339)
I20260924 04:40:20.180495 22774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b6292b3f31940ca86e72fcb7c569e13"
I20260924 04:40:20.180709 22774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:20.181996 22774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3bb3ef95c8c442e98e665bc4eac0242 in term 1.
I20260924 04:40:20.180320 22876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe: No bootstrap required, opened a new log
I20260924 04:40:20.182354 22676 leader_election.cc:302] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6292b3f31940ca86e72fcb7c569e13, d3bb3ef95c8c442e98e665bc4eac0242; no voters:
I20260924 04:40:20.182610 22887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:20.182861 22887 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [term 1 LEADER]: Becoming Leader. State: Replica: d3bb3ef95c8c442e98e665bc4eac0242, State: Running, Role: LEADER
I20260924 04:40:20.183207 22887 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } }
I20260924 04:40:20.186640 22876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } }
I20260924 04:40:20.186849 22876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfb578c4ef14fc3b0cd69d15a2de4fe, State: Initialized, Role: FOLLOWER
I20260924 04:40:20.187134 22876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } }
I20260924 04:40:20.187565 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.187736 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:20.188302 22876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:20.188337 22849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe"
I20260924 04:40:20.188553 22849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:20.189821 22849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3bb3ef95c8c442e98e665bc4eac0242 in term 1.
W20260924 04:40:20.193063 22930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:20.193202 22930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:40:20.193600 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bb3ef95c8c442e98e665bc4eac0242. Latest consensus state: current_term: 1 leader_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.193737 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:20.194347 22932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:20.195333 22932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:20.200238 22849 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [term 1 FOLLOWER]: Refusing update from remote peer d3bb3ef95c8c442e98e665bc4eac0242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:40:20.200800 22917 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:40:20.205516 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bb3ef95c8c442e98e665bc4eac0242. Latest consensus state: current_term: 1 leader_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.205634 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:20.207613 22774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [term 1 FOLLOWER]: Refusing update from remote peer d3bb3ef95c8c442e98e665bc4eac0242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:40:20.208272 22917 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:40:20.212361 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bb3ef95c8c442e98e665bc4eac0242. Latest consensus state: current_term: 1 leader_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.212503 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:20.215704 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.215842 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:20.217186 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.217322 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:20.217733 22935 mvcc.cc:205] Tried to move back new op lower bound from 7332760863533203456 to 7332760863474180096. Current Snapshot: MvccSnapshot[applied={T|T < 7332760863533203456}]
I20260924 04:40:20.220597 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.195312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.0.98.61:45203)
0924 04:40:20.195424 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.195428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.205402 (+ 9974us) server_negotiation.cc:407] Connection header received
0924 04:40:20.205586 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.205590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.205641 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.205690 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.206633 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.207391 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.208553 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.208893 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.208962 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.214902 (+ 5940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.214906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.214909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.214936 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.218780 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.219255 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.219259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.219261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.219345 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.219642 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.219645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.219647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.219855 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.219954 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.220302 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.220423 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:40:20.223546 22887 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.223672 22887 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:20.224133 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.194292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.0.98.60:45339)
0924 04:40:20.194405 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.194411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.197717 (+ 3306us) server_negotiation.cc:407] Connection header received
0924 04:40:20.197891 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.197895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.197939 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.197987 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.199196 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.199901 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.202605 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.202908 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.202983 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.213775 (+ 10792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.213779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.213782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.213810 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.220413 (+ 6603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.222496 (+ 2083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.222499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.222501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.222588 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.222884 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.222887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.222889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.223107 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.223220 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.223876 (+ 656us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.223996 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:40:20.224686 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bb3ef95c8c442e98e665bc4eac0242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35365 } } peers { permanent_uuid: "8b6292b3f31940ca86e72fcb7c569e13" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "7dfb578c4ef14fc3b0cd69d15a2de4fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45339 } } }
I20260924 04:40:20.224800 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:20.225487 22939 mvcc.cc:205] Tried to move back new op lower bound from 7332760863533203456 to 7332760863474180096. Current Snapshot: MvccSnapshot[applied={T|T < 7332760863533203456}]
I20260924 04:40:20.226025 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:20.194708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.0.98.62:35365)
0924 04:40:20.194810 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.194814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.201245 (+ 6431us) server_negotiation.cc:407] Connection header received
0924 04:40:20.201418 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.201422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.201469 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.201531 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.203491 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.204194 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.205275 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:20.205643 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:20.205723 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:20.208619 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:20.208626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:20.208630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:20.208656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:20.212014 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.212623 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.212627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.212629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.212721 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:20.220545 (+ 7824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:20.220548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:20.220550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:20.220760 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:20.220889 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:20.225770 (+ 4881us) server_negotiation.cc:299] Negotiation successful
0924 04:40:20.225890 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:40:20.226707 22932 catalog_manager.cc:1347] Generated new cluster ID: 7a4fe5bb9b1f4d8290abdfc58200e2a0
I20260924 04:40:20.226763 22932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:20.245218 22932 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:20.246697 22932 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:20.263049 22932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3bb3ef95c8c442e98e665bc4eac0242: Generated new TSK 0
I20260924 04:40:20.263873 22932 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:40:20.297358 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.294328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.0.98.62:35365)
0924 04:40:20.294447 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.294452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.296268 (+ 1816us) server_negotiation.cc:407] Connection header received
0924 04:40:20.296456 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.296459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.296516 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.296568 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.297220 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56340: BlockingRecv error: recv got EOF from 127.0.0.1:56340 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 04:40:20.298833 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.295006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.0.98.61:45203)
0924 04:40:20.295119 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.295124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.297727 (+ 2603us) server_negotiation.cc:407] Connection header received
0924 04:40:20.298088 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.298091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.298140 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.298189 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.298718 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58188: BlockingRecv error: recv got EOF from 127.0.0.1:58188 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 04:40:20.300081 22882 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.295645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.0.98.60:45339)
0924 04:40:20.295751 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.295755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.298012 (+ 2257us) server_negotiation.cc:407] Connection header received
0924 04:40:20.299343 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.299347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.299396 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.299447 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.299952 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35200: BlockingRecv error: recv got EOF from 127.0.0.1:35200 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 04:40:20.304517 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.303071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.0.98.61:45203)
0924 04:40:20.303170 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.303174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.303604 (+ 430us) server_negotiation.cc:407] Connection header received
0924 04:40:20.303810 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.303813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.303882 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.303932 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.304411 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58204: BlockingRecv error: recv got EOF from 127.0.0.1:58204 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 04:40:20.309298 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.307837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.98.61:45203)
0924 04:40:20.307934 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.307938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.308435 (+ 497us) server_negotiation.cc:407] Connection header received
0924 04:40:20.308623 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.308626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.308671 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.308719 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.309208 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58214: BlockingRecv error: recv got EOF from 127.0.0.1:58214 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 04:40:20.314613 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.313108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.0.98.61:45203)
0924 04:40:20.313273 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.313277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.313783 (+ 506us) server_negotiation.cc:407] Connection header received
0924 04:40:20.313953 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.313956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.314001 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.314049 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.314521 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58220: BlockingRecv error: recv got EOF from 127.0.0.1:58220 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260924 04:40:20.321941 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.320416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.0.98.61:45203)
0924 04:40:20.320550 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.320554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.321058 (+ 504us) server_negotiation.cc:407] Connection header received
0924 04:40:20.321262 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.321267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.321318 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.321365 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.321846 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58232: BlockingRecv error: recv got EOF from 127.0.0.1:58232 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 04:40:20.333720 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.331998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.0.98.61:45203)
0924 04:40:20.332109 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.332115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.332678 (+ 563us) server_negotiation.cc:407] Connection header received
0924 04:40:20.332869 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.332875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.332926 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.332987 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.333576 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58242: BlockingRecv error: recv got EOF from 127.0.0.1:58242 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:40:20.353914 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.352080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.0.98.61:45203)
0924 04:40:20.352206 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.352211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.352833 (+ 622us) server_negotiation.cc:407] Connection header received
0924 04:40:20.353057 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.353060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.353116 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.353196 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.353795 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58258: BlockingRecv error: recv got EOF from 127.0.0.1:58258 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:40:20.389626 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.388231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.0.98.61:45203)
0924 04:40:20.388351 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.388356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.388588 (+ 232us) server_negotiation.cc:407] Connection header received
0924 04:40:20.388802 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.388823 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.388882 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.388982 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.389506 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58274: BlockingRecv error: recv got EOF from 127.0.0.1:58274 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 04:40:20.457415 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.456153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.0.98.61:45203)
0924 04:40:20.456318 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.456322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.456460 (+ 138us) server_negotiation.cc:407] Connection header received
0924 04:40:20.456641 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.456644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.456700 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.456792 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.457300 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58286: BlockingRecv error: recv got EOF from 127.0.0.1:58286 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 04:40:20.588840 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.587678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.0.98.61:45203)
0924 04:40:20.587851 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.587856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.588075 (+ 219us) server_negotiation.cc:407] Connection header received
0924 04:40:20.588192 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.588195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.588242 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.588321 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.588744 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58294: BlockingRecv error: recv got EOF from 127.0.0.1:58294 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 04:40:20.848330 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:20.847240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.0.98.61:45203)
0924 04:40:20.847391 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:20.847395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:20.847540 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:40:20.847660 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:20.847663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:20.847708 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:20.847789 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:20.848199 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58296: BlockingRecv error: recv got EOF from 127.0.0.1:58296 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:40:21.169049 22909 catalog_manager.cc:1259] Loaded cluster ID: 7a4fe5bb9b1f4d8290abdfc58200e2a0
I20260924 04:40:21.169183 22909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13: loading cluster ID for follower catalog manager: success
I20260924 04:40:21.171789 22909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13: acquiring CA information for follower catalog manager: success
I20260924 04:40:21.173442 22909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b6292b3f31940ca86e72fcb7c569e13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:40:21.194463 22930 catalog_manager.cc:1259] Loaded cluster ID: 7a4fe5bb9b1f4d8290abdfc58200e2a0
I20260924 04:40:21.194545 22930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe: loading cluster ID for follower catalog manager: success
I20260924 04:40:21.196090 22930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe: acquiring CA information for follower catalog manager: success
I20260924 04:40:21.197160 22930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7dfb578c4ef14fc3b0cd69d15a2de4fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:40:21.366429 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:21.362618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.0.98.61:45203)
0924 04:40:21.362865 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:21.362870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:21.363107 (+ 237us) server_negotiation.cc:407] Connection header received
0924 04:40:21.363223 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:21.363225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:21.363346 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:21.363452 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:21.363692 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.364204 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.364986 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.365300 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.365379 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:21.365627 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:21.366052 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:21.366146 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:21.366266 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 04:40:21.366313 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":61,"threads_started":1}
I20260924 04:40:21.372740 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:21.369318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.98.60:45339)
0924 04:40:21.369578 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:21.369581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:21.369594 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:40:21.369684 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:21.369686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:21.369786 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:21.369888 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:21.370087 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.370570 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.371336 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.371541 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.371612 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:21.371889 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:21.372267 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:21.372369 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:21.372505 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 04:40:21.372571 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260924 04:40:21.379232 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:21.375814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.98.60:45339)
0924 04:40:21.375932 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:21.375936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:21.375949 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:40:21.375996 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:21.375999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:21.376050 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:21.376132 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:21.376411 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.377100 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.378149 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.378409 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.378511 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:21.378781 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:21.378847 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:21.378911 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:21.379051 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:40:21.379108 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:40:21.379803 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:21.375637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.0.98.62:35365)
0924 04:40:21.376042 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:21.376046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:21.376059 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:40:21.376149 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:21.376152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:21.376317 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:21.376434 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:21.376649 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.377311 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.378270 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.378608 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.378718 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:21.379059 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:21.379367 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:21.379444 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:21.379579 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 04:40:21.379630 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":62,"threads_started":1}
I20260924 04:40:21.379958 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:21.375744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.0.98.61:45203)
0924 04:40:21.375944 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:21.375948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:21.376174 (+ 226us) server_negotiation.cc:407] Connection header received
0924 04:40:21.376346 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:21.376350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:21.376412 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:21.376475 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:21.376832 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.377594 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.378709 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:21.379041 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:21.379133 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:21.379483 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:21.379556 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:21.379603 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:21.379771 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 04:40:21.379829 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260924 04:40:21.389947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 22657
I20260924 04:40:21.396387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 22732
I20260924 04:40:21.401962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 22807
2026-09-24T04:40:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3062 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:40:21 dist-test-slave-jlnt krb5kdc[23026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:40:21 dist-test-slave-jlnt krb5kdc[23026](info): set up 2 sockets
Sep 24 04:40:21 dist-test-slave-jlnt krb5kdc[23026](info): commencing operation
krb5kdc: starting...
W20260924 04:40:22.433606 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/test-user.keytab.
Sep 24 04:40:22 dist-test-slave-jlnt krb5kdc[23026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:40:22.556856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36753
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36753,127.0.98.61:32817,127.0.98.60:33489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:40:22.662776 23039 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:40:22.663069 23039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:22.663092 23039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:22.663127 23039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:22.667280 23039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:22.667343 23039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:22.667366 23039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:22.667382 23039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:22.667397 23039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:22.672587 23039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36753,127.0.98.61:32817,127.0.98.60:33489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:22.673576 23039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:22.674504 23039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:40:22.682674 23039 server_base.cc:1034] running on GCE node
W20260924 04:40:22.682616 23044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:22.682647 23045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:22.682605 23047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:22.683413 23039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:22.684404 23039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:22.685556 23039 hybrid_clock.cc:648] HybridClock initialized: now 1790224822685539 us; error 28 us; skew 500 ppm
Sep 24 04:40:22 dist-test-slave-jlnt krb5kdc[23026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:22.688119 23039 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:40:22.688868 23039 webserver.cc:460] Webserver started at http://127.0.98.62:40905/ using document root <none> and password file <none>
I20260924 04:40:22.689427 23039 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:22.689488 23039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:22.689641 23039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:22.691591 23039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/data/instance:
uuid: "34d646f69718433a82ad824154c25099"
format_stamp: "Formatted at 2026-09-24 04:40:22 on dist-test-slave-jlnt"
server_key: "fa5fbcf0b9553401358a12bd7b143e6c"
server_key_iv: "540e68b1ee97f9bb57204a94a93b1de8"
server_key_version: "encryptionkey@0"
I20260924 04:40:22.692096 23039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/wal/instance:
uuid: "34d646f69718433a82ad824154c25099"
format_stamp: "Formatted at 2026-09-24 04:40:22 on dist-test-slave-jlnt"
server_key: "fa5fbcf0b9553401358a12bd7b143e6c"
server_key_iv: "540e68b1ee97f9bb57204a94a93b1de8"
server_key_version: "encryptionkey@0"
I20260924 04:40:22.695528 23039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:40:22.697815 23054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:22.699054 23039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:40:22.699151 23039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/wal
uuid: "34d646f69718433a82ad824154c25099"
format_stamp: "Formatted at 2026-09-24 04:40:22 on dist-test-slave-jlnt"
server_key: "fa5fbcf0b9553401358a12bd7b143e6c"
server_key_iv: "540e68b1ee97f9bb57204a94a93b1de8"
server_key_version: "encryptionkey@0"
I20260924 04:40:22.699230 23039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:22.714641 23039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:22.717604 23039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:22.717752 23039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:22.724217 23039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36753
I20260924 04:40:22.724504 23106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36753 every 8 connection(s)
I20260924 04:40:22.725222 23039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:40:22.727854 23107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:22.731381 23107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:22.732259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 23039
I20260924 04:40:22.732357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-0/wal/instance
I20260924 04:40:22.732657 392 external_mini_cluster.cc:1428] Setting key d07596da937f1e2b1fa03897513e1446
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Sep 24 04:40:22 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 04:40:22.746351 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.733579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.0.98.62:36753)
0924 04:40:22.734117 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:22.734129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:22.734175 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:40:22.735146 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:22.735150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:22.735324 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:22.735511 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:22.736282 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.737129 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.738113 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.738466 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.738641 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:22.741059 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:22.741070 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:22.741081 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:22.741101 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:22.743182 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:22.743788 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:22.743794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:22.743797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:22.743939 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:22.744238 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:22.744244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:22.744249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:22.744750 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:22.744899 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:22.745336 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 04:40:22.745570 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":414,"spinlock_wait_cycles":9600,"thread_start_us":153,"threads_started":1}
I20260924 04:40:22.746356 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.733579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36753 (local address 127.0.0.1:50608)
0924 04:40:22.734014 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.734065 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:22.734845 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:22.735496 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:22.735501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:22.735916 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:22.736140 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:22.736152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.737304 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.737307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:22.737998 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:22.738006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.738614 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.738618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:22.738723 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:22.739323 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:22.739338 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:22.740895 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:22.743308 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:22.743317 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:22.743327 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:22.743667 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:22.744037 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:22.744039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:22.744041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:22.744146 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:22.744886 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:22.744893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:22.745070 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 04:40:22.745283 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 04:40:22.745678 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":84,"thread_start_us":121,"threads_started":1}
I20260924 04:40:22.748916 23107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:22.749648 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.749414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32817 (local address 127.0.0.1:51804)
0924 04:40:22.749520 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.749588 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:40:22.749965 23055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
I20260924 04:40:22.750790 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.750603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32817 (local address 127.0.0.1:51808)
0924 04:40:22.750706 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.750746 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:40:22.751024 23107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32817: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:40:22.760360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32817
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36753,127.0.98.61:32817,127.0.98.60:33489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:40:22.800267 23107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } attempt: 1
I20260924 04:40:22.801241 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.800872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32817 (local address 127.0.0.1:51814)
0924 04:40:22.801042 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.801106 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:40:22.801975 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.801745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32817 (local address 127.0.0.1:51828)
0924 04:40:22.801857 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.801913 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:40:22.802134 23107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32817: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
W20260924 04:40:22.868432 23114 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:40:22.868727 23114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:22.868753 23114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:22.868791 23114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:40:22.870345 23107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } attempt: 2
I20260924 04:40:22.871134 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.870848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32817 (local address 127.0.0.1:51836)
0924 04:40:22.870996 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.871062 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 04:40:22.871891 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.871670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32817 (local address 127.0.0.1:51848)
0924 04:40:22.871777 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.871830 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:40:22.872067 23107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32817: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32817: connect: Connection refused (error 111)
W20260924 04:40:22.873075 23114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:22.873121 23114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:22.873157 23114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:22.873174 23114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:22.873189 23114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:22.878298 23114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36753,127.0.98.61:32817,127.0.98.60:33489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:22.879220 23114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:22.880110 23114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:40:22.888142 23114 server_base.cc:1034] running on GCE node
W20260924 04:40:22.888121 23122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:22.888098 23120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:22.888098 23119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:22.888849 23114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:22.890206 23114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:22.891409 23114 hybrid_clock.cc:648] HybridClock initialized: now 1790224822891389 us; error 32 us; skew 500 ppm
Sep 24 04:40:22 dist-test-slave-jlnt krb5kdc[23026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:22.894235 23114 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:40:22.895048 23114 webserver.cc:460] Webserver started at http://127.0.98.61:38909/ using document root <none> and password file <none>
I20260924 04:40:22.895587 23114 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:22.895678 23114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:22.895898 23114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:22.897924 23114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/data/instance:
uuid: "d168659a397642078598b8e7fa20a468"
format_stamp: "Formatted at 2026-09-24 04:40:22 on dist-test-slave-jlnt"
server_key: "a020c96777bdb7eb5779c7a2285c48e6"
server_key_iv: "1157e9be7fe0d781b362acbde7e33984"
server_key_version: "encryptionkey@0"
I20260924 04:40:22.898483 23114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/wal/instance:
uuid: "d168659a397642078598b8e7fa20a468"
format_stamp: "Formatted at 2026-09-24 04:40:22 on dist-test-slave-jlnt"
server_key: "a020c96777bdb7eb5779c7a2285c48e6"
server_key_iv: "1157e9be7fe0d781b362acbde7e33984"
server_key_version: "encryptionkey@0"
I20260924 04:40:22.902078 23114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:40:22.904425 23129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:22.905658 23114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:40:22.905788 23114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/wal
uuid: "d168659a397642078598b8e7fa20a468"
format_stamp: "Formatted at 2026-09-24 04:40:22 on dist-test-slave-jlnt"
server_key: "a020c96777bdb7eb5779c7a2285c48e6"
server_key_iv: "1157e9be7fe0d781b362acbde7e33984"
server_key_version: "encryptionkey@0"
I20260924 04:40:22.905892 23114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:22.937500 23114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:22.940274 23114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:22.940481 23114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:22.947497 23114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32817
I20260924 04:40:22.947781 23181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32817 every 8 connection(s)
I20260924 04:40:22.948457 23114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:40:22.951129 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:22.954770 23182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:22.956167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 23114
I20260924 04:40:22.956296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-1/wal/instance
I20260924 04:40:22.956568 392 external_mini_cluster.cc:1428] Setting key 8a0ae34d5d979dc17d53ed88027662cc
I20260924 04:40:22.963244 23107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Sep 24 04:40:22 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 04:40:22.969347 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.956662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.0.98.62:36753)
0924 04:40:22.956793 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:22.956797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:22.957787 (+ 990us) server_negotiation.cc:407] Connection header received
0924 04:40:22.958791 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:22.958796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:22.958867 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:22.958967 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:22.960050 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.960943 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.962114 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.962520 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.962643 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:22.965175 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:22.965179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:22.965182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:22.965204 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:22.966819 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:22.967422 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:22.967436 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:22.967441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:22.967532 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:22.967904 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:22.967907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:22.967909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:22.968151 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:22.968251 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:22.969048 (+ 797us) server_negotiation.cc:299] Negotiation successful
0924 04:40:22.969221 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 04:40:22 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:40:22.970742 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.957062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36753 (local address 127.0.0.1:50622)
0924 04:40:22.957605 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.957664 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:22.958590 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:22.959128 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:22.959135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:22.959467 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:22.959902 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:22.959914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.961088 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.961091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:22.961979 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:22.961988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.962665 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.962667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:22.962799 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:22.963451 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:22.963474 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:22.965026 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:22.966922 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:22.966927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:22.966941 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:22.967283 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:22.967622 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:22.967625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:22.967627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:22.967794 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:22.968308 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:22.968317 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:22.968636 (+ 319us) client_negotiation.cc:769] Sending connection context
0924 04:40:22.969519 (+ 883us) client_negotiation.cc:240] Negotiation successful
0924 04:40:22.969799 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":108,"threads_started":1}
I20260924 04:40:22.972898 23182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:22.975198 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.963670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32817 (local address 127.0.0.1:51864)
0924 04:40:22.963838 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.963854 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:22.963939 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:22.964679 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:22.964682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:22.964872 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:22.964978 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:22.964983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.966671 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.966674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:22.967431 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:22.967440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.968159 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.968163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:22.968281 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:22.968725 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:22.968737 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:22.971328 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:22.973487 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:22.973490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:22.973492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:22.973737 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:22.974057 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:22.974062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:22.974067 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:22.974156 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:22.974690 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:22.974693 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:22.974786 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 04:40:22.974922 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 04:40:22.975042 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 04:40:22.975384 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.963869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.0.98.61:32817)
0924 04:40:22.964200 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:22.964213 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:22.964237 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:40:22.964276 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:22.964281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:22.964483 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:22.965137 (+ 654us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:22.965288 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.966492 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.967561 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.968048 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.968160 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:22.971525 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:22.971535 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:22.971548 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:22.971574 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:22.973342 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:22.973846 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:22.973851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:22.973853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:22.973937 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:22.974262 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:22.974267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:22.974269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:22.974573 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:22.974714 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:22.975029 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 04:40:22.975174 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":138,"threads_started":1}
I20260924 04:40:22.977097 23107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:22.977846 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.977621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:43918)
0924 04:40:22.977730 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.977790 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 04:40:22.978708 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.978429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:43924)
0924 04:40:22.978580 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.978636 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 04:40:22.978848 23107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Sep 24 04:40:22 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:40:22.982589 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.973432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32817 (local address 127.0.0.1:51866)
0924 04:40:22.973532 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.973547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:22.973611 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:22.974028 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:22.974033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:22.974194 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:22.974308 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:22.974322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.975218 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.975222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:22.975985 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:22.975993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.977114 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.977118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:22.977297 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:22.977771 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:22.977791 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:22.979382 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:22.981160 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:22.981163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:22.981165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:22.981356 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:22.981636 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:22.981639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:22.981641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:22.981720 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:22.982075 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:22.982078 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:22.982154 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 04:40:22.982262 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:40:22.982396 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":42}
I20260924 04:40:22.982620 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.973475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.0.98.61:32817)
0924 04:40:22.973796 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:22.973807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:22.973821 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:40:22.973854 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:22.973857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:22.973906 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:22.973999 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:22.974421 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.975113 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.976116 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:22.977009 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:22.977122 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:22.979544 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:22.979548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:22.979550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:22.979576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:22.981013 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:22.981452 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:22.981457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:22.981459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:22.981541 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:22.981816 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:22.981819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:22.981820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:22.981979 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:22.982084 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:22.982315 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:40:22.982450 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":241,"thread_start_us":132,"threads_started":1}
I20260924 04:40:22.983312 23182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:40:22.984095 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.983866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:43936)
0924 04:40:22.983959 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.984043 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 04:40:22.984301 23131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
I20260924 04:40:22.984464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
/tmp/dist-test-taskVN1buQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33489
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36753,127.0.98.61:32817,127.0.98.60:33489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:40:22.985114 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:22.984905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:43946)
0924 04:40:22.985009 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:22.985066 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:40:22.985304 23182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
I20260924 04:40:23.009996 23107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } attempt: 1
I20260924 04:40:23.010759 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.010495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:43962)
0924 04:40:23.010633 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.010689 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 04:40:23.011431 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.011242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:43970)
0924 04:40:23.011341 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.011388 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:40:23.011548 23107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
I20260924 04:40:23.034539 23182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } attempt: 1
I20260924 04:40:23.035338 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.035073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:43978)
0924 04:40:23.035212 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.035267 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:40:23.036056 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.035827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:43988)
0924 04:40:23.035934 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.035984 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 04:40:23.036185 23182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
I20260924 04:40:23.086712 23107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } attempt: 2
I20260924 04:40:23.087630 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.087305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:44000)
0924 04:40:23.087493 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.087555 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 04:40:23.088356 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.088155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:44002)
0924 04:40:23.088259 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.088308 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:40:23.088483 23107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
W20260924 04:40:23.093317 23190 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:40:23.093607 23190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:40:23.093665 23190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:40:23.093741 23190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:40:23.097712 23190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:40:23.097780 23190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:40:23.097826 23190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:40:23.097879 23190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:40:23.097919 23190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:40:23.103609 23190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36753,127.0.98.61:32817,127.0.98.60:33489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:40:23.104386 23182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } attempt: 2
I20260924 04:40:23.104691 23190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:40:23.105203 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.104861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:44014)
0924 04:40:23.105023 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.105078 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 04:40:23.105850 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.105666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:44030)
0924 04:40:23.105762 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.105803 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 04:40:23.105988 23190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:40:23.106019 23182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33489: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33489: connect: Connection refused (error 111)
W20260924 04:40:23.113973 23196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:23.113977 23198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:40:23.114006 23195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:40:23.114187 23190 server_base.cc:1034] running on GCE node
I20260924 04:40:23.114742 23190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:40:23.115744 23190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:40:23.116905 23190 hybrid_clock.cc:648] HybridClock initialized: now 1790224823116882 us; error 32 us; skew 500 ppm
Sep 24 04:40:23 dist-test-slave-jlnt krb5kdc[23026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:40:23.119760 23190 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:40:23.120627 23190 webserver.cc:460] Webserver started at http://127.0.98.60:39775/ using document root <none> and password file <none>
I20260924 04:40:23.121210 23190 fs_manager.cc:359] Metadata directory not provided
I20260924 04:40:23.121274 23190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:40:23.121477 23190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:40:23.123576 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/data/instance:
uuid: "7c1c3e4743964b5891a3af8fd25fa989"
format_stamp: "Formatted at 2026-09-24 04:40:23 on dist-test-slave-jlnt"
server_key: "560e3ee84a7164ce4a38a6c2da204247"
server_key_iv: "eecfbd2be409c18901cde366b5c65eaf"
server_key_version: "encryptionkey@0"
I20260924 04:40:23.124143 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/wal/instance:
uuid: "7c1c3e4743964b5891a3af8fd25fa989"
format_stamp: "Formatted at 2026-09-24 04:40:23 on dist-test-slave-jlnt"
server_key: "560e3ee84a7164ce4a38a6c2da204247"
server_key_iv: "eecfbd2be409c18901cde366b5c65eaf"
server_key_version: "encryptionkey@0"
I20260924 04:40:23.127655 23190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:40:23.129984 23205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:40:23.131163 23190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:40:23.131295 23190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/wal
uuid: "7c1c3e4743964b5891a3af8fd25fa989"
format_stamp: "Formatted at 2026-09-24 04:40:23 on dist-test-slave-jlnt"
server_key: "560e3ee84a7164ce4a38a6c2da204247"
server_key_iv: "eecfbd2be409c18901cde366b5c65eaf"
server_key_version: "encryptionkey@0"
I20260924 04:40:23.131393 23190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:40:23.158108 23190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:40:23.161124 23190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:40:23.161408 23190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:40:23.168365 23190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33489
I20260924 04:40:23.168663 23257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33489 every 8 connection(s)
I20260924 04:40:23.169435 23190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:40:23.170457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu as pid 23190
I20260924 04:40:23.170583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVN1buQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790224545298817-392-0/minicluster-data/master-2/wal/instance
I20260924 04:40:23.170905 392 external_mini_cluster.cc:1428] Setting key 7c2414c2605b4ee460128ce8f00a686d
I20260924 04:40:23.173125 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:40:23.178294 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:40:23 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:40:23 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:40:23.187651 23107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } attempt: 3
I20260924 04:40:23.188999 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.172787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.0.98.60:33489)
0924 04:40:23.174071 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.174090 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.174130 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:40:23.175115 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.175133 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.175591 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.175974 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.177454 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.178984 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.180055 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.180721 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.180868 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.183504 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.183508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.183516 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.183541 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.185397 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.185837 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.185842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.185850 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.185968 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.186605 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.186608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.186610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.187181 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.187389 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.187640 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.188006 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":158,"threads_started":1}
I20260924 04:40:23.189618 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.180253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.0.98.62:36753)
0924 04:40:23.180380 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.180386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.180660 (+ 274us) server_negotiation.cc:407] Connection header received
0924 04:40:23.180818 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.180821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.180872 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.180947 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.181931 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.182447 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.183360 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.183644 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.183722 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.186080 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.186083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.186085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.186108 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.187717 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.188268 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.188271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.188272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.188354 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.188652 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.188655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.188657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.188848 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.188918 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.189404 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.189517 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:40:23.190240 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.180181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36753 (local address 127.0.0.1:50634)
0924 04:40:23.180541 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.180578 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:23.180708 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:23.181116 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:23.181124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:23.181608 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:23.181790 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.181800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.182572 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.182574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.183226 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.183235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.183760 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.183766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.183883 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.184308 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:23.184327 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:23.185943 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:23.187838 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.187845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.187850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.188151 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.188465 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.188468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.188470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.188551 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.189011 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:23.189019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:23.189176 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 04:40:23.189852 (+ 676us) client_negotiation.cc:240] Negotiation successful
0924 04:40:23.190062 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":115,"threads_started":1}
I20260924 04:40:23.191283 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:40:23 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:40:23 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:40:23.197192 23182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } attempt: 3
I20260924 04:40:23.197404 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.188056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:44050)
0924 04:40:23.188170 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.188187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:23.188258 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:23.188725 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:23.188728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:23.188883 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:23.188954 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.188958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.190009 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.190014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.190810 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.190820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.191780 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.191784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.191869 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.192320 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:23.192333 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:23.193789 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:23.195681 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.195684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.195686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.195939 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.196259 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.196262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.196264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.196359 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.196828 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:23.196832 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:23.196936 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 04:40:23.197066 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 04:40:23.197217 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
Sep 24 04:40:23 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:40:23.198088 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.188244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.0.98.60:33489)
0924 04:40:23.188540 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.188546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.188556 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:40:23.188582 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.188584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.188615 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.188708 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.189068 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.189826 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.191306 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.191653 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.192073 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.193920 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.193925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.193929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.193958 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.195585 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.196058 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.196062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.196064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.196143 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.196501 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.196505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.196508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.196724 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.196821 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.197126 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.197287 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":140,"threads_started":1}
I20260924 04:40:23.201385 23107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099: Bootstrap starting.
I20260924 04:40:23.201836 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.192318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.0.98.61:32817)
0924 04:40:23.192483 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.192488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.192503 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:40:23.192544 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.192547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.192590 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.192670 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.193250 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.193901 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.195381 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.195672 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.195789 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.198217 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.198223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.198228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.198253 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.200069 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.200515 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.200520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.200522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.200607 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.200910 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.200916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.200921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.201094 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.201209 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.201564 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.201682 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:40:23.202288 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.191668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32817 (local address 127.0.0.1:51878)
0924 04:40:23.192125 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.192139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:23.192207 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:23.192779 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:23.192781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:23.192984 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:23.193077 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.193084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.194015 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.194018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.195256 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.195265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.195772 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.195774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.195841 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.196248 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:23.196260 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:23.198089 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:23.200185 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.200189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.200191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.200430 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.200705 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.200708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.200710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.200798 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.201286 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:23.201288 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:23.201367 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 04:40:23.202022 (+ 655us) client_negotiation.cc:240] Negotiation successful
0924 04:40:23.202139 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385}
I20260924 04:40:23.202906 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:40:23.203112 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.191725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.0.98.62:36753)
0924 04:40:23.191824 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.191834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.191843 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:40:23.191867 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.191869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.191898 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.191991 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.192706 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.193321 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.194446 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.194780 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.194874 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.198594 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.198598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.198600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.198624 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.200334 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.200774 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.200777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.200779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.200857 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.201175 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.201181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.201184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.201373 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.202708 (+ 1335us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.202788 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.202952 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 04:40:23 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:40:23.205161 23107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:23.206127 23107 log.cc:824] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:23.208352 23107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099: No bootstrap required, opened a new log
I20260924 04:40:23.208920 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.197816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.0.98.60:33489)
0924 04:40:23.197942 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.197946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.197958 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:40:23.197990 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.197992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.198048 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.198399 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.198753 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.201841 (+ 3088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.202875 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.203180 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.203607 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.205522 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.205525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.205528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.205550 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.207152 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.207631 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.207634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.207636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.207711 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.208012 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.208017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.208019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.208206 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.208299 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.208623 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.208742 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:40:23.209249 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.197587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:44060)
0924 04:40:23.197828 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.197842 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:23.197917 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:23.198184 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:23.198190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:23.198530 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:23.198636 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.198644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.201971 (+ 3327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.201974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.202724 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.202740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.203309 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.203312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.203398 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.203857 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:23.203870 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:23.205356 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:23.207272 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.207275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.207277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.207497 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.207815 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.207817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.207819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.207901 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.208294 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:23.208300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:23.208423 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 04:40:23.208545 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 04:40:23.208693 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
Sep 24 04:40:23 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:40:23 dist-test-slave-jlnt krb5kdc[23026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790224823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:40:23.212405 23107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } }
I20260924 04:40:23.212644 23107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d646f69718433a82ad824154c25099, State: Initialized, Role: FOLLOWER
I20260924 04:40:23.212642 23182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468: Bootstrap starting.
I20260924 04:40:23.212971 23107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } }
I20260924 04:40:23.214567 23107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:23.215405 23182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:23.215871 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.203664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.0.98.60:33489)
0924 04:40:23.204719 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.204724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.204739 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:40:23.204774 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.204777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.204816 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.204895 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.205487 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.206315 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.207343 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.207687 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.207766 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.211549 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.211556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.211562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.211588 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.213834 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.214258 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.214261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.214263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.214348 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.214705 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.214710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.214714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.214886 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.214965 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.215605 (+ 640us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.215716 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983}
I20260924 04:40:23.215935 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.204297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.0.98.61:32817)
0924 04:40:23.204393 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.204397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.204407 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:40:23.204443 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.204445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.204481 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.204566 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.205333 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.206572 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.207596 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.207899 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.208007 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.211598 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.211602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.211604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.211629 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.213716 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.214251 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.214254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.214256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.214348 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.214666 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.214670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.214672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.214881 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.215070 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.215692 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.215793 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:40:23.216338 23182 log.cc:824] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:23.216408 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.203530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33489 (local address 127.0.0.1:44072)
0924 04:40:23.203894 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 04:40:23.203908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:40:23.203995 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:40:23.205047 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:40:23.205050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:40:23.205291 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:40:23.205374 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.205380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.206429 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.206432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.207231 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:40:23.207240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.207853 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.207856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:40:23.207924 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.209798 (+ 1874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:40:23.209812 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:40:23.211390 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:40:23.213935 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.213941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.213945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.214165 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.214442 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:40:23.214444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:40:23.214446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:40:23.214544 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:40:23.215004 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:40:23.215006 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:40:23.215090 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 04:40:23.215197 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:40:23.215334 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299}
I20260924 04:40:23.219487 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989: Bootstrap starting.
I20260924 04:40:23.219568 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:40:23.221129 23182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468: No bootstrap required, opened a new log
W20260924 04:40:23.221114 23276 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:23.221238 23276 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:40:23.222463 23258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989: Neither blocks nor log segments found. Creating new log.
I20260924 04:40:23.223311 23258 log.cc:824] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989: Log is configured to *not* fsync() on all Append() calls
I20260924 04:40:23.217119 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
I20260924 04:40:23.223702 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 04:40:23.219098 23060 tablet.cc:2369] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:40:23.220731 23081 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "34d646f69718433a82ad824154c25099"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642
I20260924 04:40:23.224373 23081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:40:23.224479 23081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:23.224370 23182 raft_consensus.cc:348] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } }
I20260924 04:40:23.224603 23182 raft_consensus.cc:729] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d168659a397642078598b8e7fa20a468, State: Initialized, Role: FOLLOWER
I20260924 04:40:23.224931 23182 consensus_queue.cc:260] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } }
I20260924 04:40:23.225435 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989: No bootstrap required, opened a new log
I20260924 04:40:23.225800 23081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } }
I20260924 04:40:23.226219 23182 sys_catalog.cc:564] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:23.226729 23081 leader_election.cc:288] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [CANDIDATE]: Term 1 election: Requested vote from peers d168659a397642078598b8e7fa20a468 (127.0.98.61:32817), 7c1c3e4743964b5891a3af8fd25fa989 (127.0.98.60:33489)
I20260924 04:40:23.229367 23258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } }
I20260924 04:40:23.229591 23258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1c3e4743964b5891a3af8fd25fa989, State: Initialized, Role: FOLLOWER
I20260924 04:40:23.229638 23156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34d646f69718433a82ad824154c25099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d168659a397642078598b8e7fa20a468"
I20260924 04:40:23.229856 23156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:23.229915 23258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } }
I20260924 04:40:23.231122 23258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:40:23.231138 23156 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d646f69718433a82ad824154c25099 in term 1.
I20260924 04:40:23.231412 23232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34d646f69718433a82ad824154c25099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7c1c3e4743964b5891a3af8fd25fa989"
I20260924 04:40:23.231748 23232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:40:23.233070 23232 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d646f69718433a82ad824154c25099 in term 1.
I20260924 04:40:23.233345 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
I20260924 04:40:23.233464 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:23.233531 23056 leader_election.cc:302] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d646f69718433a82ad824154c25099, 7c1c3e4743964b5891a3af8fd25fa989; no voters:
I20260924 04:40:23.234076 23265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:40:23.234283 23265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [term 1 LEADER]: Becoming Leader. State: Replica: 34d646f69718433a82ad824154c25099, State: Running, Role: LEADER
I20260924 04:40:23.234607 23265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } }
I20260924 04:40:23.236413 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d646f69718433a82ad824154c25099. Latest consensus state: current_term: 1 leader_uuid: "34d646f69718433a82ad824154c25099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
W20260924 04:40:23.236796 23307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:23.236876 23307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 04:40:23.240075 23295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:40:23.240147 23295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:40:23.240896 23114 master_runner.cc:263] No leader master found from master d168659a397642078598b8e7fa20a468
I20260924 04:40:23.240965 23114 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:40:23.230289 23039 master_runner.cc:186] Error getting master registration for 127.0.98.61:32817: OK, instance_id { permanent_uuid: "d168659a397642078598b8e7fa20a468" instance_seqno: 1790224822945653 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:40:23.244254 23039 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:40:23.244180 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:23.245464 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
I20260924 04:40:23.245591 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:23.251772 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.233904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.0.98.61:32817)
0924 04:40:23.234030 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.234035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.237344 (+ 3309us) server_negotiation.cc:407] Connection header received
0924 04:40:23.242481 (+ 5137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.242484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.242627 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.242682 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.243610 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.244351 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.245828 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.246181 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.246250 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.247990 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.247995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.248001 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.248028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.249804 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.250385 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.250389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.250392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.250477 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.250797 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.250800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.250804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.251044 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.251148 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.251493 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.251601 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:40:23.252524 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.235059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.0.98.60:33489)
0924 04:40:23.238106 (+ 3047us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.238110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.238324 (+ 214us) server_negotiation.cc:407] Connection header received
0924 04:40:23.238793 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.238796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.238843 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.238893 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.242857 (+ 3964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.243533 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.246104 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.246457 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.246579 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.248189 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.248195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.248201 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.248230 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.250544 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.250977 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.250980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.250982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.251055 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.251353 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.251358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.251363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.251583 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.251702 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.252257 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.252371 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2969}
I20260924 04:40:23.253499 23315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:40:23.254941 23315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:40:23.256996 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:23.235594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.0.98.62:36753)
0924 04:40:23.235774 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.235778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.238526 (+ 2748us) server_negotiation.cc:407] Connection header received
0924 04:40:23.239047 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.239050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.239096 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.239147 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.240975 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.241714 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.244981 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:23.245313 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:23.245390 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:23.253511 (+ 8121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:40:23.253518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:40:23.253522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:40:23.253559 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:40:23.255235 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.255702 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.255706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.255708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.255792 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:40:23.256104 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:40:23.256107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:40:23.256108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:40:23.256301 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:40:23.256397 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:23.256735 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 04:40:23.256857 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260924 04:40:23.266315 23232 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [term 1 FOLLOWER]: Refusing update from remote peer 34d646f69718433a82ad824154c25099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:40:23.266975 23308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:40:23.270607 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d646f69718433a82ad824154c25099. Latest consensus state: current_term: 1 leader_uuid: "34d646f69718433a82ad824154c25099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
I20260924 04:40:23.270718 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:23.273588 23156 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [term 1 FOLLOWER]: Refusing update from remote peer 34d646f69718433a82ad824154c25099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:40:23.274103 23308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:40:23.277854 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d646f69718433a82ad824154c25099. Latest consensus state: current_term: 1 leader_uuid: "34d646f69718433a82ad824154c25099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
I20260924 04:40:23.277968 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:23.283090 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34d646f69718433a82ad824154c25099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
I20260924 04:40:23.283221 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:23.284343 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34d646f69718433a82ad824154c25099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
I20260924 04:40:23.284456 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:23.285733 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34d646f69718433a82ad824154c25099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
I20260924 04:40:23.285845 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099 [sys.catalog]: This master's current role is: LEADER
I20260924 04:40:23.286835 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34d646f69718433a82ad824154c25099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d646f69718433a82ad824154c25099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } peers { permanent_uuid: "d168659a397642078598b8e7fa20a468" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32817 } } peers { permanent_uuid: "7c1c3e4743964b5891a3af8fd25fa989" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33489 } } }
I20260924 04:40:23.286943 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:40:23.287101 23316 mvcc.cc:205] Tried to move back new op lower bound from 7332760876091138048 to 7332760875972915200. Current Snapshot: MvccSnapshot[applied={T|T < 7332760876091138048}]
I20260924 04:40:23.287298 23317 mvcc.cc:205] Tried to move back new op lower bound from 7332760876091138048 to 7332760875972915200. Current Snapshot: MvccSnapshot[applied={T|T < 7332760876091138048}]
I20260924 04:40:23.287463 23319 mvcc.cc:205] Tried to move back new op lower bound from 7332760876091138048 to 7332760875972915200. Current Snapshot: MvccSnapshot[applied={T|T < 7332760876091138048}]
I20260924 04:40:23.289081 23315 catalog_manager.cc:1347] Generated new cluster ID: 915810bbc0ef4182812604e6d557268e
I20260924 04:40:23.289168 23315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:40:23.313230 23315 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:40:23.314567 23315 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:40:23.335251 23315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34d646f69718433a82ad824154c25099: Generated new TSK 0
I20260924 04:40:23.336197 23315 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:40:23.372514 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.369080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.0.98.61:32817)
0924 04:40:23.369239 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.369244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.371194 (+ 1950us) server_negotiation.cc:407] Connection header received
0924 04:40:23.371609 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.371614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.371665 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.371721 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.372375 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51914: BlockingRecv error: recv got EOF from 127.0.0.1:51914 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 04:40:23.373795 23263 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.369545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.0.98.60:33489)
0924 04:40:23.369673 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.369678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.371400 (+ 1722us) server_negotiation.cc:407] Connection header received
0924 04:40:23.372860 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.372863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.372926 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.372978 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.373675 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44098: BlockingRecv error: recv got EOF from 127.0.0.1:44098 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 04:40:23.375339 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.369913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.0.98.62:36753)
0924 04:40:23.370038 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.370043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.374277 (+ 4234us) server_negotiation.cc:407] Connection header received
0924 04:40:23.374640 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.374643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.374697 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.374748 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.375229 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50662: BlockingRecv error: recv got EOF from 127.0.0.1:50662 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 04:40:23.379773 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.378154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.0.98.61:32817)
0924 04:40:23.378255 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.378259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.378781 (+ 522us) server_negotiation.cc:407] Connection header received
0924 04:40:23.379032 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.379037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.379130 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.379192 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.379679 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51916: BlockingRecv error: recv got EOF from 127.0.0.1:51916 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:40:23.384667 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.383125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.0.98.61:32817)
0924 04:40:23.383219 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.383223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.383771 (+ 548us) server_negotiation.cc:407] Connection header received
0924 04:40:23.383990 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.383995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.384053 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.384111 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.384576 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51926: BlockingRecv error: recv got EOF from 127.0.0.1:51926 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 04:40:23.390225 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.388741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.0.98.61:32817)
0924 04:40:23.388835 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.388839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.389347 (+ 508us) server_negotiation.cc:407] Connection header received
0924 04:40:23.389531 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.389534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.389581 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.389631 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.390136 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51940: BlockingRecv error: recv got EOF from 127.0.0.1:51940 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 04:40:23.398134 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.396345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.0.98.61:32817)
0924 04:40:23.396476 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.396480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.396972 (+ 492us) server_negotiation.cc:407] Connection header received
0924 04:40:23.397208 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.397211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.397262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.397317 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.398045 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51944: BlockingRecv error: recv got EOF from 127.0.0.1:51944 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 04:40:23.409768 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.408267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.0.98.61:32817)
0924 04:40:23.408376 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.408380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.408850 (+ 470us) server_negotiation.cc:407] Connection header received
0924 04:40:23.409022 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.409026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.409069 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.409129 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.409679 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51956: BlockingRecv error: recv got EOF from 127.0.0.1:51956 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 04:40:23.429966 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.428264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.0.98.61:32817)
0924 04:40:23.428380 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.428384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.428904 (+ 520us) server_negotiation.cc:407] Connection header received
0924 04:40:23.429101 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.429105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.429205 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.429259 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.429792 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51960: BlockingRecv error: recv got EOF from 127.0.0.1:51960 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 04:40:23.465759 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.464583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.0.98.61:32817)
0924 04:40:23.464733 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.464738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.464923 (+ 185us) server_negotiation.cc:407] Connection header received
0924 04:40:23.465035 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.465038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.465096 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.465215 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.465647 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51968: BlockingRecv error: recv got EOF from 127.0.0.1:51968 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 04:40:23.533277 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.531935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.0.98.61:32817)
0924 04:40:23.532109 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.532115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.532278 (+ 163us) server_negotiation.cc:407] Connection header received
0924 04:40:23.532453 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.532457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.532508 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.532662 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.533135 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51972: BlockingRecv error: recv got EOF from 127.0.0.1:51972 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 04:40:23.664688 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.663425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.0.98.61:32817)
0924 04:40:23.663613 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.663618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.663747 (+ 129us) server_negotiation.cc:407] Connection header received
0924 04:40:23.663937 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.663941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.663997 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.664089 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.664583 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51980: BlockingRecv error: recv got EOF from 127.0.0.1:51980 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 04:40:23.924079 23188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:40:23.922933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.98.61:32817)
0924 04:40:23.923072 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:23.923077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:23.923188 (+ 111us) server_negotiation.cc:407] Connection header received
0924 04:40:23.923369 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:23.923372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:23.923427 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:23.923517 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:40:23.923968 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51996: BlockingRecv error: recv got EOF from 127.0.0.1:51996 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:40:24.238117 23307 catalog_manager.cc:1259] Loaded cluster ID: 915810bbc0ef4182812604e6d557268e
I20260924 04:40:24.238198 23307 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989: loading cluster ID for follower catalog manager: success
I20260924 04:40:24.239794 23307 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989: acquiring CA information for follower catalog manager: success
I20260924 04:40:24.241153 23307 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7c1c3e4743964b5891a3af8fd25fa989: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:40:24.241333 23295 catalog_manager.cc:1259] Loaded cluster ID: 915810bbc0ef4182812604e6d557268e
I20260924 04:40:24.241390 23295 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468: loading cluster ID for follower catalog manager: success
I20260924 04:40:24.242725 23295 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468: acquiring CA information for follower catalog manager: success
I20260924 04:40:24.243628 23295 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d168659a397642078598b8e7fa20a468: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:40:24.442513 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:24.438051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.0.98.61:32817)
0924 04:40:24.438297 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:24.438302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:24.438420 (+ 118us) server_negotiation.cc:407] Connection header received
0924 04:40:24.438597 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:24.438601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:24.438733 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:24.438854 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:24.439132 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.439659 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.440870 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.441320 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.441422 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:24.441705 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:24.442136 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:24.442229 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:24.442277 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:40:24.442324 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260924 04:40:24.448714 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:24.445434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.0.98.60:33489)
0924 04:40:24.445653 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:24.445656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:24.445667 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:40:24.445776 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:24.445778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:24.445933 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:24.446079 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:24.446241 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.446723 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.447468 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.447676 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.447746 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:24.447971 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:24.448348 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:24.448433 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:24.448482 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:40:24.448527 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260924 04:40:24.455588 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:24.451841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.0.98.61:32817)
0924 04:40:24.451942 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:24.451966 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:24.452093 (+ 127us) server_negotiation.cc:407] Connection header received
0924 04:40:24.452194 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:24.452197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:24.452250 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:24.452321 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:24.452902 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.453449 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.454422 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.454704 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.454811 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:24.455117 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:24.455195 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:24.455264 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:24.455414 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 04:40:24.455463 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:40:24.455588 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:24.451679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.0.98.60:33489)
0924 04:40:24.451815 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:24.451820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:24.452189 (+ 369us) server_negotiation.cc:407] Connection header received
0924 04:40:24.452355 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:24.452358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:24.452399 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:24.452476 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:24.452830 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.453489 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.454503 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.454826 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.454920 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:24.455173 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:24.455258 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:24.455333 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:24.455412 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 04:40:24.455463 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:40:24.456903 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:40:24.451641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.0.98.62:36753)
0924 04:40:24.452174 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 04:40:24.452178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:40:24.452193 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:40:24.452274 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:40:24.452276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:40:24.452439 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:40:24.452548 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:40:24.453238 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.454024 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.455161 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:40:24.455517 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:40:24.455613 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:40:24.455925 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:40:24.456254 (+ 329us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:40:24.456338 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:40:24.456724 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 04:40:24.456794 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":69,"threads_started":1}
I20260924 04:40:24.460040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 23039
I20260924 04:40:24.466574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 23114
I20260924 04:40:24.472385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVN1buQ/build/debug/bin/kudu with pid 23190
2026-09-24T04:40:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3070 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6133 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279166 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST